diff --git a/README.md b/README.md index fa0d8395..a0b9e869 100644 --- a/README.md +++ b/README.md @@ -14,11 +14,11 @@ Equip your LLM chatbot with a powerful and flexible long term memory system. [![](https://img.shields.io/badge/Contribute-Welcome-green)](https://modelscope.github.io/memoryscope/tutorial/contribute.html) ---- -## News +## 📰 News - **[2024-09-06]** We release MemoryScope v0.1.1 now, which is also available in [PyPI](https://pypi.org/simple/memoryscope/)! ---- -## What is MemoryScope？ +## 🌟 What is MemoryScope？ MemoryScope provides LLM chatbots with powerful and flexible long-term memory capabilities, offering a framework for building such abilities. It can be applied to scenarios like personal assistants and emotional companions, continuously learning through long-term memory to remember users' basic information as well as various habits and preferences. This allows users to gradually experience a sense of "understanding" when using the LLM. @@ -84,8 +84,8 @@ In the future, we will support more model interfaces and local deployment of LLM For installation, please refer to [Installation.md](docs/installation.md). -## Example Usages -- [Simple Usages (Quick Start)](./examples/api/simple_usages_en.ipynb) +## 🍕 Example Usages +- [Simple Usages (Quick Start)](./examples/api/simple_usages.ipynb) - [With AutoGen](./examples/api/autogen_example.md) - [CLI with a MemoryScope Chatbot](./examples/cli/README.md) - [Advanced Customization](./examples/advance/custom_operator.md) @@ -102,7 +102,7 @@ poetry install --with dev pre-commit install ``` -Please refer to our [Contribution Guide](./docs/contribute.md) for more details. +Please refer to our [Contribution Guide](./docs/contribution.md) for more details. ## 📖 Citation diff --git a/README_ZH.md b/README_ZH.md index bf1fc6f4..3f42ab35 100644 --- a/README_ZH.md +++ b/README_ZH.md @@ -19,12 +19,12 @@ ---- -## 新闻 +## 📰 新闻 - **[2024-09-06]** 我们现在发布了 MemoryScope v0.1.1，该版本也可以在 [PyPI](https://pypi.org/simple/memoryscope/) 上获取！ ---- -## 什么是MemoryScope？ +## 🌟 什么是MemoryScope？ MemoryScope可以为LLM聊天机器人提供强大且灵活的长期记忆能力，并提供了构建长期记忆能力的框架。 MemoryScope可以用于个人助理、情感陪伴等记忆场景，通过长期记忆能力来不断学习，记得用户的基础信息以及各种习惯和喜好，使得用户在使用LLM时逐渐感受到一种“默契”。 @@ -78,25 +78,10 @@ MemoryScope可以用于个人助理、情感陪伴等记忆场景，通过长期 ---- ## 🚀 安装 - -### 1. 使用 Docker 安装 - -### 2. 使用 Docker Compose 安装 - -### 3. 使用 PYPI + Docker 安装 [仅限 Linux & MacOS] - -### 3. 使用 从源码安装 + Docker 安装 [仅限 Linux & MacOS] - -### Docker方式一键运行Demo - - - 完整的安装方法请参考[安装指南](docs/installation_zh.md)。 -## 快速开始 -- [简易用法（快速开始）](./examples/api/simple_usages_cn.ipynb) +## 🍕 快速开始 +- [简易用法（快速开始）](./examples/api/simple_usages_zh.ipynb) - [在命令行与MemoryScope聊天机器人交互](./examples/cli/README_ZH.md) - [进阶自定义用法](./examples/advance/custom_operator.md) - [结合AutoGen使用](./examples/api/autogen_example.md) @@ -110,7 +95,7 @@ MemoryScope可以用于个人助理、情感陪伴等记忆场景，通过长期 pip install -e . pre-commit install ``` -请参阅我们的[贡献指南](./docs/contribute_zh.md) 了解更多详细信息。 +请参阅我们的[贡献指南](./docs/contribution_zh.md) 了解更多详细信息。 ## 📖 引用 diff --git a/docs/sphinx_doc/en/source/docs/contribute.md b/docs/contribution.md similarity index 100% rename from docs/sphinx_doc/en/source/docs/contribute.md rename to docs/contribution.md diff --git a/docs/sphinx_doc/zh_CN/source/docs/contribute_zh.md b/docs/contribution_zh.md similarity index 100% rename from docs/sphinx_doc/zh_CN/source/docs/contribute_zh.md rename to docs/contribution_zh.md diff --git a/docs/sphinx_doc/en/source/docs/installation.md b/docs/installation.md similarity index 99% rename from docs/sphinx_doc/en/source/docs/installation.md rename to docs/installation.md index ea8dab8e..413214a3 100644 --- a/docs/sphinx_doc/en/source/docs/installation.md +++ b/docs/installation.md @@ -114,7 +114,7 @@ The docker method is recommended: docker.elastic.co/elasticsearch/elasticsearch:8.13.2 ``` -4. Launch memoryscope, also refer to [cli documents](../examples/cli/README.md) +4. Launch memoryscope, also refer to [cli documents](../examples/cli/CLI_README.md) ```bash export OPENAI_API_KEY="sk-0000000000" python quick-start-demo.py --config_path=memoryscope/core/config/demo_config_zh.yaml diff --git a/docs/sphinx_doc/zh_CN/source/docs/installation_zh.md b/docs/installation_zh.md similarity index 99% rename from docs/sphinx_doc/zh_CN/source/docs/installation_zh.md rename to docs/installation_zh.md index c58c67a7..41d29f46 100644 --- a/docs/sphinx_doc/zh_CN/source/docs/installation_zh.md +++ b/docs/installation_zh.md @@ -115,7 +115,7 @@ docker.elastic.co/elasticsearch/elasticsearch:8.13.2 ``` -4. 启动 MemoryScope，同时参考 [CLI 文档](../examples/cli/README.md) +4. 启动 MemoryScope，同时参考 [CLI 文档](../examples/cli/CLI_README_ZH.md) ```bash export DASHSCOPE_API_KEY="sk-0000000000" python quick-start-demo.py --config_path=memoryscope/core/config/demo_config_zh.yaml diff --git a/docs/sphinx_doc/Makefile b/docs/sphinx_doc/Makefile index bc8097ab..dc1c479b 100644 --- a/docs/sphinx_doc/Makefile +++ b/docs/sphinx_doc/Makefile @@ -6,8 +6,8 @@ ASSETSDIR = assets BUILDDIR = build/html SOURCEDIR_EN = en/source BUILDDIR_EN = build/html/en -SOURCEDIR_ZH = zh_CN/source -BUILDDIR_ZH = build/html/zh_CN +SOURCEDIR_ZH = zh/source +BUILDDIR_ZH = build/html/zh # English document en: @@ -16,7 +16,7 @@ en: @echo "Build finished. The HTML pages are in $(BUILDDIR_EN)" # Chinese document -zh_CN: +zh: @$(SPHINXBUILD) -b html "$(SOURCEDIR_ZH)" "$(BUILDDIR_ZH)" @echo @echo "Build finished. The HTML pages are in $(BUILDDIR_ZH)" @@ -27,6 +27,6 @@ index: %: Makefile @$(SPHINXBUILD) -M $@ "$(SOURCEDIR_EN)" "$(BUILDDIR_EN)" $(O) -all: en zh_CN index +all: en zh index -.PHONY: all en zh_CN index \ No newline at end of file +.PHONY: all en zh index \ No newline at end of file diff --git a/docs/sphinx_doc/build_sphinx_doc.sh b/docs/sphinx_doc/build_sphinx_doc.sh index cce54cc2..172b0b41 100755 --- a/docs/sphinx_doc/build_sphinx_doc.sh +++ b/docs/sphinx_doc/build_sphinx_doc.sh @@ -1,11 +1,9 @@ #!/bin/bash -cd docs/sphinx_doc - # remove build rm -rf build/html/* -rm en/source/memoryscope*.rst -rm zh_CN/source/memoryscope*.rst +rm -rf en/source/memoryscope*.rst +rm -rf zh/source/memoryscope*.rst # copy related files @@ -13,29 +11,32 @@ cd ../../ cp README.md docs/sphinx_doc/en/source/README.md cp docs/installation.md docs/sphinx_doc/en/source/docs/installation.md +cp docs/contribution.md docs/sphinx_doc/en/source/docs/contribution.md cp -r docs/images docs/sphinx_doc/en/source/docs/images cp -r examples docs/sphinx_doc/en/source/examples -cp README_ZH.md docs/sphinx_doc/zh_CN/source/README.md -cp docs/installation_ZH.md docs/sphinx_doc/zh_CN/source/docs/installation.md -cp -r docs/images docs/sphinx_doc/zh_CN/source/docs/images -cp -r examples docs/sphinx_doc/zh_CN/source/examples - +cp README_ZH.md docs/sphinx_doc/zh/source/README.md +cp docs/installation_zh.md docs/sphinx_doc/zh/source/docs/installation.md +cp docs/contribution_zh.md docs/sphinx_doc/zh/source/docs/contribution.md +cp -r docs/images docs/sphinx_doc/zh/source/docs/images +cp -r examples docs/sphinx_doc/zh/source/examples # build cd docs/sphinx_doc sphinx-apidoc -f -o en/source ../../memoryscope -t template -e -sphinx-apidoc -f -o zh_CN/source ../../memoryscope -t template -e +sphinx-apidoc -f -o zh/source ../../memoryscope -t template -e # clear redundant files make clean all rm en/source/README.md rm en/source/docs/installation.md +rm en/source/docs/contribution.md rm -rf en/source/docs/images rm -rf en/source/examples -rm zh_CN/source/README.md -rm zh_CN/source/docs/installation.md -rm -rf zh_CN/source/docs/images -rm -rf zh_CN/source/examples \ No newline at end of file +rm zh/source/README.md +rm zh/source/docs/installation.md +rm zh/source/docs/contribution.md +rm -rf zh/source/docs/images +rm -rf zh/source/examples \ No newline at end of file diff --git a/docs/sphinx_doc/en/source/_templates/language_selector.html b/docs/sphinx_doc/en/source/_templates/language_selector.html index a8aca93e..86fe0703 100644 --- a/docs/sphinx_doc/en/source/_templates/language_selector.html +++ b/docs/sphinx_doc/en/source/_templates/language_selector.html @@ -1,5 +1,5 @@
English | - 中文 + 中文
diff --git a/docs/sphinx_doc/en/source/index.rst b/docs/sphinx_doc/en/source/index.rst index 2352f596..825b4c4e 100644 --- a/docs/sphinx_doc/en/source/index.rst +++ b/docs/sphinx_doc/en/source/index.rst @@ -14,25 +14,26 @@ Welcome to MemoryScope Tutorial .. image:: docs/images/logo.png :align: center -MemoryScope is a powerful and flexible long term memory system for LLM chatbots. It consists -of a memory database and three customizable system operations, which can be flexibly combined to provide -robust long term memory services for your LLM chatbot. +MemoryScope provides LLM chatbots with powerful and flexible long-term memory capabilities, offering a framework for building such abilities. +It can be applied to scenarios like personal assistants and emotional companions, continuously learning through long-term memory to remember users' basic information as well as various habits and preferences. +This allows users to gradually experience a sense of "understanding" when using the LLM. -💾 Memory Database: +.. image:: docs/images/framework.png + :align: center + +Framework ^^^^^^^^^^^^^^^^^^^^ -- MemoryScope comes with an *ElasticSearch (ES)* vector database to store all the -memory pieces recorded in the system. +💾 Memory Database: MemoryScope is equipped with a vector database (default is *ElasticSearch*) to store all memory fragments recorded in the system. -🛠️ System operations: -^^^^^^^^^^^^^^^^^^^^^ +🔧 Worker Library: MemoryScope atomizes the capabilities of long-term memory into individual workers, including over 20 workers for tasks such as query information filtering, observation extraction, and insight updating. + +🛠️ Operation Library: Based on the worker pipeline, it constructs the operations for memory services, realizing key capabilities such as memory retrieval and memory consolidation. - Memory Retrieval: Upon arrival of a user query, this operation returns the semantically related memory pieces and/or those from the corresponding time if the query involves reference to time. - - Memory Consolidation: This operation takes in a batch of user queries and returns important user information extracted from the queries as consolidated *observations* to be stored in the memory database. - - Reflection and Re-consolidation: At regular intervals, this operation performs reflection upon newly recorded *observations* to form and update *insights*. Then, memory re-consolidation is performed to ensure contradictions and repetitions among memory pieces are properly handled. @@ -42,9 +43,12 @@ among memory pieces are properly handled. :caption: MemoryScope Tutorial About MemoryScope - 🚀 Installation - Cli Client - Simple Usages + Installation + Cli Client + Simple Usages + Advanced usage + Contribution + .. toctree:: :maxdepth: 6 diff --git a/docs/sphinx_doc/requirements.txt b/docs/sphinx_doc/requirements.txt index d0763f76..d71d0830 100644 --- a/docs/sphinx_doc/requirements.txt +++ b/docs/sphinx_doc/requirements.txt @@ -10,4 +10,5 @@ sphinx_rtd_theme sphinxcontrib-mermaid myst-parser autodoc_pydantic -nbsphinx \ No newline at end of file +nbsphinx +Pandoc \ No newline at end of file diff --git a/docs/sphinx_doc/zh_CN/source/_static/custom.css b/docs/sphinx_doc/zh/source/_static/custom.css similarity index 100% rename from docs/sphinx_doc/zh_CN/source/_static/custom.css rename to docs/sphinx_doc/zh/source/_static/custom.css diff --git a/docs/sphinx_doc/zh_CN/source/_templates/language_selector.html b/docs/sphinx_doc/zh/source/_templates/language_selector.html similarity index 67% rename from docs/sphinx_doc/zh_CN/source/_templates/language_selector.html rename to docs/sphinx_doc/zh/source/_templates/language_selector.html index a8aca93e..86fe0703 100644 --- a/docs/sphinx_doc/zh_CN/source/_templates/language_selector.html +++ b/docs/sphinx_doc/zh/source/_templates/language_selector.html @@ -1,5 +1,5 @@
English | - 中文 + 中文
diff --git a/docs/sphinx_doc/zh_CN/source/_templates/layout.html b/docs/sphinx_doc/zh/source/_templates/layout.html similarity index 100% rename from docs/sphinx_doc/zh_CN/source/_templates/layout.html rename to docs/sphinx_doc/zh/source/_templates/layout.html diff --git a/docs/sphinx_doc/zh_CN/source/conf.py b/docs/sphinx_doc/zh/source/conf.py similarity index 99% rename from docs/sphinx_doc/zh_CN/source/conf.py rename to docs/sphinx_doc/zh/source/conf.py index 2b8f00ad..b6cf426b 100644 --- a/docs/sphinx_doc/zh_CN/source/conf.py +++ b/docs/sphinx_doc/zh/source/conf.py @@ -19,7 +19,7 @@ sys.path.insert(0, os.path.abspath("../../../../../MemoryScope")) # -- Project information ----------------------------------------------------- -language = "zh_CN" +language = "zh" project = "MemoryScope" copyright = "2024, Alibaba Tongyi Lab" diff --git a/docs/sphinx_doc/zh_CN/source/docs/api.rst b/docs/sphinx_doc/zh/source/docs/api.rst similarity index 100% rename from docs/sphinx_doc/zh_CN/source/docs/api.rst rename to docs/sphinx_doc/zh/source/docs/api.rst diff --git a/docs/sphinx_doc/zh/source/index.rst b/docs/sphinx_doc/zh/source/index.rst new file mode 100644 index 00000000..138f696f --- /dev/null +++ b/docs/sphinx_doc/zh/source/index.rst @@ -0,0 +1,53 @@ +.. MemoryScope documentation master file, created by + sphinx-quickstart on Fri Jan 5 17:53:54 2024. + You can adapt this file completely to your liking, but it should at least + contain the root `toctree` directive. + +:github_url: https://github.com/modelscope/memoryscope + +MemoryScope 文档 +========================= + +欢迎浏览MemoryScope相关文档 +------------------------------- + +.. image:: ./docs/images/logo.png + :align: center + +MemoryScope可以为LLM聊天机器人提供强大且灵活的长期记忆能力，并提供了构建长期记忆能力的框架。 +MemoryScope可以用于个人助理、情感陪伴等记忆场景，通过长期记忆能力来不断学习，记得用户的基础信息以及各种习惯和喜好，使得用户在使用LLM时逐渐感受到一种“默契”。 + +.. image:: docs/images/framework.png + :align: center + +核心框架 +^^^^^^^^^^^^^^^^^^^^ + +💾 记忆数据库: MemoryScope配备了向量数据库(默认是*ElasticSearch*)，用于存储系统中记录的所有记忆片段。 + +🔧 核心worker库: MemoryScope将长期记忆的能力原子化，抽象成单独的worker，包括query信息过滤，observation抽取，insight更新等20+worker。 + +🛠️ 核心Op库: 并基于worker的pipeline构建了memory服务的核心operation，实现了记忆检索，记忆巩固等核心能力。 + +- 记忆检索：当用户输入对话，此操作返回语义相关的记忆片段。如果输入对话包含对时间的指涉，则同时返回相应时间中的记忆片段。 +- 记忆巩固：此操作接收一批用户的输入对话，并从对话中提取重要的用户信息，将其作为 *observation* 形式的记忆片段存储在记忆数据库中。 +- 反思与再巩固：每隔一段时间，此操作对新记录的 *observations* 进行反思，以形成和更新 *insight* + 形式的记忆片段。然后执行记忆再巩固，以确保记忆片段之间的矛盾和重复得到妥善处理。 + +.. toctree:: + :maxdepth: 2 + :caption: MemoryScope 教程 + + 关于 MemoryScope + 安装 + 命令行终端 + 简单案例 + 高级用法 + 贡献 + +.. toctree:: + :maxdepth: 6 + :caption: MemoryScope 接口 + + API + diff --git a/docs/sphinx_doc/zh_CN/source/modules.rst b/docs/sphinx_doc/zh/source/modules.rst similarity index 100% rename from docs/sphinx_doc/zh_CN/source/modules.rst rename to docs/sphinx_doc/zh/source/modules.rst diff --git a/docs/sphinx_doc/zh_CN/source/index.rst b/docs/sphinx_doc/zh_CN/source/index.rst deleted file mode 100644 index 5301c8d5..00000000 --- a/docs/sphinx_doc/zh_CN/source/index.rst +++ /dev/null @@ -1,49 +0,0 @@ -.. MemoryScope documentation master file, created by - sphinx-quickstart on Fri Jan 5 17:53:54 2024. - You can adapt this file completely to your liking, but it should at least - contain the root `toctree` directive. - -:github_url: https://github.com/modelscope/memoryscope - -MemoryScope 文档 -========================= - -欢迎浏览MemoryScope相关文档 -------------------------------- - -.. image:: ./docs/images/logo_1.png - :align: center - -MemoryScope 是一个为LLM聊天机器人服务的强大且灵活的长期记忆系统。它由一个记忆数据库和三个可定制的系统操作组成，这些操作可以灵活组合， -为您的LLM聊天机器人提供强大的长期记忆服务。 - -💾 记忆数据库: -^^^^^^^^^^^^^ - -- MemoryScope 配备了一个 *ElasticSearch (ES)* 向量数据库，用于存储系统中记录的所有记忆片段。 - -🛠️ 系统操作: -^^^^^^^^^^^^ - -- 记忆检索：当用户输入对话，此操作返回语义相关的记忆片段。如果输入对话包含对时间的指涉，则同时返回相应时间中的记忆片段。 -- -- 记忆巩固：此操作接收一批用户的输入对话，并从对话中提取重要的用户信息，将其作为 *observation* 形式的记忆片段存储在记忆数据库中。 -- -- 反思与再巩固：每隔一段时间，此操作对新记录的 *observations* 进行反思，以形成和更新 *insight* 形式的记忆片段。然后执行记忆再巩固， -以确保记忆片段之间的矛盾和重复得到妥善处理。 - -.. toctree:: - :maxdepth: 2 - :caption: MemoryScope 教程 - - 关于MemoryScope - 🚀 安装 - 命令行 - 简例 - -.. toctree:: - :maxdepth: 6 - :caption: MemoryScope 接口 - - API - diff --git a/examples/api/simple_usages_en.ipynb b/examples/api/simple_usages.ipynb similarity index 97% rename from examples/api/simple_usages_en.ipynb rename to examples/api/simple_usages.ipynb index 1b4ea574..4a9cde70 100644 --- a/examples/api/simple_usages_en.ipynb +++ b/examples/api/simple_usages.ipynb @@ -24,8 +24,6 @@ "metadata": {}, "outputs": [], "source": [ - "import sys\n", - "sys.path.append(\".\")\n", "from memoryscope import MemoryScope, Arguments\n", "arguments = Arguments(\n", " language=\"en\",\n", @@ -481,9 +479,9 @@ "metadata": {}, "source": [ "## More Examples\n", - "We direct the reader to [Advanced Customization](./examples/api/advanced_customization_en.ipynb) for guidance on customizing the various settings of the MemoryScope system. It is also possible to create or customize your own MemoryScope **operations** by specifying a **workflow** and the corresponding **workers** that best meet your specific needs.\n", + "We direct the reader to [Advanced Customization](../advance/custom_operator.md) for guidance on customizing the various settings of the MemoryScope system. It is also possible to create or customize your own MemoryScope **operations** by specifying a **workflow** and the corresponding **workers** that best meet your specific needs.\n", "\n", - "Additionally, you can also try out the [CLI with a MemoryScope Chatbot](../cli/README.md). We have implemented the chatbot so that the **Memory Consolidation** and **Reflection and Re-Consolidation** operations are always run asynchronously in the backend, ensuring that they do not incur any response time for the user.\n", + "Additionally, you can also try out the [CLI with a MemoryScope Chatbot](../cli/CLI_README.md). We have implemented the chatbot so that the **Memory Consolidation** and **Reflection and Re-Consolidation** operations are always run asynchronously in the backend, ensuring that they do not incur any response time for the user.\n", "\n" ] } diff --git a/examples/api/simple_usages_cn.ipynb b/examples/api/simple_usages_zh.ipynb similarity index 98% rename from examples/api/simple_usages_cn.ipynb rename to examples/api/simple_usages_zh.ipynb index fd7ef019..a47c97ef 100644 --- a/examples/api/simple_usages_cn.ipynb +++ b/examples/api/simple_usages_zh.ipynb @@ -459,9 +459,9 @@ "metadata": {}, "source": [ "## 更多用法\n", - "我们建议读者参考[进阶自定义用法](../advance/custom_operator.md)来对MemoryScope系统进行各种自定义设置。您还可以通过自定义**workflow**和对应的**worker**来创建或定制满足您特定需求的**operation**。\n", + "我们建议读者参考[进阶自定义用法](../advance/custom_operator_zh.md)来对MemoryScope系统进行各种自定义设置。您还可以通过自定义**workflow**和对应的**worker**来创建或定制满足您特定需求的**operation**。\n", "\n", - "此外，您还可以尝试使用[在命令行与MemoryScope聊天机器人交互](../cli/README_ZH.md)。我们在这里实现了始终在后台异步运行**记忆巩固**和**反思与再巩固**这两个操作，从而使得它们不会增加聊天的响应时间。" + "此外，您还可以尝试使用[在命令行与MemoryScope聊天机器人交互](../cli/CLI_README_ZH.md)。我们在这里实现了始终在后台异步运行**记忆巩固**和**反思与再巩固**这两个操作，从而使得它们不会增加聊天的响应时间。" ] } ],