ReMe/doc/vector_store_setup.md

🚀 Vector Store Quick Start Guide

This comprehensive guide covers all available vector store implementations in ExperienceMaker, their differences, use cases, and setup instructions.

📋 Overview

ExperienceMaker supports multiple vector store backends for different use cases and deployment scenarios:

• FileVectorStore (backend=local_file) - 📁 Local file-based storage for development and small datasets
• ChromaVectorStore (backend=chroma) - 🔮 Embedded vector database for local development and moderate scale
• EsVectorStore (backend=elasticsearch) - 🔍 Elasticsearch-based storage for production and large scale

⚡ Vector Store Implementations

1. 📁 FileVectorStore (backend=local_file)

A simple file-based vector store that saves data to local JSONL files. Perfect for development, testing, and small datasets.

💡 When to Use

• Development and testing - No external dependencies required 🛠️
• Small datasets - Suitable for datasets with < 10,000 vectors 📊
• Single-user applications - No concurrent access support 👤
• Prototyping - Quick setup without infrastructure ⚡

✨ Features

• ✅ No external dependencies
• ✅ Simple file-based persistence
• ✅ Built-in cosine similarity search
• ❌ No concurrent access support
• ❌ Limited scalability
• ❌ No advanced filtering

⚙️ Configuration Parameters

from experiencemaker.vector_store import FileVectorStore
from experiencemaker.embedding_model.openai_compatible_embedding_model import OpenAICompatibleEmbeddingModel

embedding_model = OpenAICompatibleEmbeddingModel(dimensions=1024, model_name="text-embedding-v4")

vector_store = FileVectorStore(
    embedding_model=embedding_model,
    store_dir="./file_vector_store",  # Directory to store JSONL files
    batch_size=1024                   # Batch size for operations
)

💻 Example Usage

# Create workspace and insert data
workspace_id = "my_workspace"
vector_store.create_workspace(workspace_id)

nodes = [
    VectorNode(
        workspace_id=workspace_id,
        content="Artificial intelligence is revolutionizing technology",
        metadata={"category": "tech", "source": "article1"}
    ),
    VectorNode(
        workspace_id=workspace_id,
        content="Machine learning enables data-driven insights",
        metadata={"category": "tech", "source": "article2"}
    )
]

vector_store.insert(nodes, workspace_id)

# Search
results = vector_store.search("What is AI?", workspace_id, top_k=2)

2. 🔮 ChromaVectorStore (backend=chroma)

An embedded vector database that provides persistent storage with advanced features while remaining easy to deploy.

💡 When to Use

• Local development with persistence requirements 🏠
• Medium-scale applications (10K - 1M vectors) 📈
• Multi-user applications with moderate concurrency 👥
• Applications requiring metadata filtering 🔍
• Docker deployments without external database dependencies 🐳

✨ Features

• ✅ Persistent embedded database
• ✅ Advanced metadata filtering
• ✅ Built-in vector indexing (HNSW)
• ✅ HTTP API support
• ✅ Concurrent access support
• ✅ Collection management
• ❌ Limited horizontal scaling

⚙️ Configuration Parameters

from experiencemaker.vector_store import ChromaVectorStore
from experiencemaker.embedding_model.openai_compatible_embedding_model import OpenAICompatibleEmbeddingModel

embedding_model = OpenAICompatibleEmbeddingModel(dimensions=1024, model_name="text-embedding-v4")

vector_store = ChromaVectorStore(
    embedding_model=embedding_model,
    store_dir="./chroma_vector_store",  # Directory for Chroma database
    batch_size=1024                     # Batch size for operations
)

💻 Example Usage

workspace_id = "chroma_workspace"

# Check if workspace exists
if not vector_store.exist_workspace(workspace_id):
    vector_store.create_workspace(workspace_id)

# Insert with metadata
nodes = [
    VectorNode(
        workspace_id=workspace_id,
        content="Deep learning models require large datasets",
        metadata={"category": "AI", "difficulty": "advanced", "topic": "deep_learning"}
    )
]

vector_store.insert(nodes, workspace_id)

# Search with results
results = vector_store.search("deep learning", workspace_id, top_k=5)
for result in results:
    print(f"Content: {result.content}")
    print(f"Metadata: {result.metadata}")

3. 🔍 EsVectorStore (backend=elasticsearch)

Production-grade vector search using Elasticsearch with advanced filtering, scaling, and enterprise features.

💡 When to Use

• Production environments requiring high availability 🏭
• Large-scale applications (1M+ vectors) 🚀
• High-throughput scenarios with many concurrent users ⚡
• Complex filtering requirements on metadata 🎯
• Distributed deployments across multiple nodes 🌐
• Enterprise environments with existing Elasticsearch infrastructure 🏢

✨ Features

• ✅ Horizontal scaling
• ✅ High availability and fault tolerance
• ✅ Advanced filtering and aggregations
• ✅ Real-time indexing and search
• ✅ Cluster management
• ✅ Enterprise security features
• ✅ Monitoring and analytics
• ❌ Complex setup and maintenance
• ❌ Higher resource requirements

🛠️ Setup Elasticsearch

Before using EsVectorStore, you need to set up Elasticsearch. Choose one of the following methods:

Option 1: All-in-One Script (Recommended for Development) 🎯

curl -fsSL https://elastic.co/start-local | sh

Option 2: Docker Run with HTTP Host 🐳

# Pull the latest Elasticsearch image
docker pull docker.elastic.co/elasticsearch/elasticsearch-wolfi:9.0.0

# Run Elasticsearch container
docker run -p 9200:9200 \
  -e "discovery.type=single-node" \
  -e "xpack.security.enabled=false" \
  -e "xpack.license.self_generated.type=trial" \
  -e "http.host=0.0.0.0" \
  docker.elastic.co/elasticsearch/elasticsearch-wolfi:9.0.0

🔧 Environment Configuration

Set the Elasticsearch hosts environment variable:

export ES_HOSTS=http://localhost:9200

⚙️ Configuration Parameters

from experiencemaker.vector_store import EsVectorStore
from experiencemaker.embedding_model.openai_compatible_embedding_model import OpenAICompatibleEmbeddingModel
import os

embedding_model = OpenAICompatibleEmbeddingModel(dimensions=1024, model_name="text-embedding-v4")

vector_store = EsVectorStore(
    embedding_model=embedding_model,
    hosts=os.getenv("ES_HOSTS", "http://localhost:9200"),  # Elasticsearch hosts
    basic_auth=None,                                       # ("username", "password") for auth
    batch_size=1024,                                      # Batch size for bulk operations
    retrieve_filters=[]                                   # Pre-configured filters
)

🎯 Advanced Filtering

EsVectorStore supports advanced filtering capabilities:

# Add term filters
vector_store.add_term_filter("metadata.category", "technology")
vector_store.add_term_filter("metadata.language", "en")

# Add range filters
vector_store.add_range_filter("metadata.score", gte=0.8)
vector_store.add_range_filter("metadata.timestamp", gte="2024-01-01", lte="2024-12-31")

# Search with filters applied
results = vector_store.search("machine learning", workspace_id, top_k=10)

# Clear filters for next search
vector_store.clear_filter()

💻 Example Usage

from experiencemaker.schema.vector_node import VectorNode

# Configure connection
workspace_id = "production_workspace"

# Create workspace with custom mapping
if not vector_store.exist_workspace(workspace_id):
    vector_store.create_workspace(workspace_id)

# Insert with rich metadata
nodes = [
    VectorNode(
        workspace_id=workspace_id,
        content="Transformer architecture revolutionized NLP",
        metadata={
            "category": "AI",
            "subcategory": "NLP",
            "author": "research_team",
            "timestamp": "2024-01-15",
            "confidence": 0.95,
            "tags": ["transformer", "nlp", "attention"]
        }
    )
]

# Insert with refresh for immediate availability
vector_store.insert(nodes, workspace_id, refresh=True)

# Advanced search with filters
vector_store.add_term_filter("metadata.category", "AI")
vector_store.add_range_filter("metadata.confidence", gte=0.9)

results = vector_store.search("transformer models", workspace_id, top_k=5)

for result in results:
    print(f"Score: {result.metadata.get('_score', 'N/A')}")
    print(f"Content: {result.content}")
    print(f"Metadata: {result.metadata}")

🎉 This guide provides everything you need to get started with vector stores in ExperienceMaker. Choose the implementation that best fits your use case and scale up as needed! ✨