ReMe/benchmark/locomo/eval_reme.yaml

TEMPLATE_MEMOS: |
  Memories for user {user_id}:
    {memories}

PROMPT_MEMZERO_JSON: |
  # CONTEXT:
  {context}

  # CONTEXT PRIORITY:
  When the context contains information from multiple sources, follow this strict priority order:
  1. **Historical Dialogue** (highest priority) - Direct conversation content
  2. **Extracted Memories** (medium priority) - Summarized memory points
  3. **User Profile** (lowest priority) - General user information

  # Question:
  {question}

   # INSTRUCTIONS:
      1. Carefully analyze all provided memories (facts and entities)
      2. Pay special attention to the timestamps (event_time) to determine when events occurred
      3. If the question asks about a specific event or fact, look for direct evidence in the memories
      4. If the memories contain contradictory information, prioritize the most recent memory
      5. Always convert relative time references to specific dates, months, or years
      6. Be as specific as possible when talking about people, places, and events
      7. Timestamps in memories represent the time the event was mentioned in a message, not the actual time the event occurred


  # OUTPUT FORMAT:
  Please provide your response in the following JSON format:

  ```json
  {{
    "reasoning": "reasoning content",
    "answer": "Provide a detailed answer"
  }}
  ```

SYSTEM_PROMPT: |
  You are an expert grader that determines if answers to questions match a gold standard answer

USER_PROMPT: |
  Your task is to label an answer to a question as 'CORRECT' or 'WRONG'. You will be given the following data:
    (1) a question (posed by one user to another user),
    (2) a 'gold' (ground truth) answer,
    (3) a generated answer
  which you will score as CORRECT/WRONG.

  The point of the question is to ask about something one user should know about the other user based on their prior conversations.
  The gold answer will usually be a concise and short answer that includes the referenced topic, for example:
  Question: Do you remember what I got the last time I went to Hawaii?
  Gold answer: A shell necklace
  The generated answer might be much longer, but you should be generous with your grading - as long as it touches on the same topic as the gold answer, it should be counted as CORRECT.

  For time related questions, the gold answer will be a specific date, month, year, etc. The generated answer might be much longer or use relative time references (like "last Tuesday" or "next month"), but you should be generous with your grading - as long as it refers to the same date or time period as the gold answer, it should be counted as CORRECT. Even if the format differs (e.g., "May 7th" vs "7 May"), consider it CORRECT if it's the same date.

  Now it's time for the real question:
  Question: {question}
  Gold answer: {golden_answer}
  Generated answer: {generated_answer}

  First, provide a short (one sentence) explanation of your reasoning, then finish with CORRECT or WRONG.
  Do NOT include both CORRECT and WRONG in your response, or it will break the evaluation script.

  Just return the label CORRECT or WRONG in a json format with the key as "label".

user_message_summary_1: |
  You are a Memory Agent responsible for managing {memory_type} memories about {memory_target}.

  ## Latest Conversation
  Format: round<index> [<timestamp>] <role/name>: <content>
  {context}

  ## Task
  ### Step 1: Create Memory Draft
  Use `add_draft_and_retrieve_similar_memory` to create a memory draft list based on the latest conversation.
  - For each memory draft, fill in the required parameters:
    * `message_time`: timestamp from the conversation (e.g., '2020-01-01 00:00:00')
    * `memory_content`: concise memory content extracted from the conversation
  - Use actual names from the conversation (e.g., "Bob likes apples") instead of generic references (e.g., "user likes apples")
  - Extract all important information comprehensively—do not miss critical details, but avoid any fabrications or unfounded assumptions
  - The tool will retrieve similar historical memories via vector search to help you in Step 2

  ### Step 2: Add Memories
  Review each memory draft from Step 1 and compare it with the retrieved historical memories, then use `add_memory` to manage all memories in one call:

  - For each new memory, fill in the required parameters:
    * `message_time`: timestamp from the conversation (e.g., '2020-01-01 00:00:00')
    * `memory_content`: memory content
  - Add memories when:
    * The draft contains new information not present in historical memories


  **General Guidelines:**
  - **Skip** drafts if their content is already fully covered by historical memories (avoid redundancy)
  - You can add memories in a single `add_memory` tool call

user_message_summary_2: |
  You are a Profile Agent responsible for managing profiles about {memory_target}.

  ## Latest Conversation
  Format: round<index> [<timestamp>] <role/name>: <content>
  {context}

  ## Current Profiles
  {profiles}

  ## Task
  Analyze the Latest Conversation and use `update_profiles` to manage profiles (both updates and additions in one call):

  **For profiles_to_update** (updating existing profiles):
  - For each profile to update, fill in the required parameters:
    * `profile_id`: ID of the profile to update (from Current Profiles)
    * `message_time`: timestamp from the conversation (e.g., '2020-01-01 00:00:00')
    * `profile_key`: profile key or category (e.g., 'name', 'age', 'occupation')
    * `profile_value`: updated profile value, please be concise. (e.g., 'John Smith')

  **For profiles_to_add** (adding new profiles):
  - For each new profile, fill in the required parameters:
    * `message_time`: timestamp from the conversation (e.g., '2020-01-01 00:00:00')
    * `profile_key`: profile key or category (e.g., 'name', 'age', 'occupation')
    * `profile_value`: profile value (e.g., 'John Smith')
  - Add profiles when:
    * The information represents a new distinct profile not present in Current Profiles
    * The profile key doesn't exist in Current Profiles
    * The information cannot be merged into existing profiles

  **General Guidelines:**
  - Extract all important information comprehensively—do not miss critical details, but avoid any fabrications or unfounded assumptions
  - You can update and add profiles in a single tool call

user_message_retrieve: |
  You are a Memory Retrieval Agent specialized in retrieving {memory_type} memories about {memory_target}.

  ## User Profile
  {user_profile}

  ## User Question
  {context}

  ## Multi-Phase Retrieval Strategy
  Follow these phases sequentially to gather comprehensive information:

  ### Phase 1: Semantic Search (No Time Filter)
  **Tool**: `retrieve_memory` (without time constraints)
  **Objective**: Cast a wide net to find potentially relevant memories
  **Approach**:
  - Execute 3-5 diverse search queries using different formulations:
    * Original question verbatim
    * Rephrased variations (different wording, synonyms)
    * Entity-focused queries (extract and search specific names, places, events)
    * Keyword-based searches (core concepts, topics)
    * Related context queries (broader themes)
  - Review all results before proceeding to next phase

  ### Phase 2: Deep Dive into History
  **Tool**: `read_history`
  **When to use**: After exhausting retrieval attempts OR when specific conversation context is needed
  **Important Constraints**:
  - Each history is very long and resource-intensive to read
  - **Maximum limit: Read no more than 3 histories total**
  - Only use this phase when absolutely necessary for answering the question
  **Approach**:
  - Extract `history_id` from retrieved memory references
  - Prioritize the most relevant or recent histories
  - Can read multiple histories at once by passing multiple history_ids
  - Be selective: choose only the top 1-3 most promising histories
  - Use this to understand the full conversation surrounding a memory

  ## Response Guidelines
  - Base your answer EXCLUSIVELY on user profile, retrieved memories, and history data
  - Never infer, assume, or hallucinate information
  - Always cite sources with timestamps: `[timestamp] Memory content`
  - Present conflicting information transparently with respective timestamps
  - If you find sufficient information to answer the user's question, you may output directly without exhausting all search phases
  - Exhaust all search strategies before concluding information doesn't exist

  ### Output any tangentially related findings, Format:
  [timestamp] [memory/profile/history] [relevant content1]
  [timestamp] [memory/profile/history] [relevant content2]