ReMe/cookbook/daily_paper/README.md

Daily Paper Cookbook

中文

Daily Paper is a local-first, file-native workflow for turning research feeds into a daily reading package. It collects papers from the Hugging Face weekly and monthly rankings, removes yesterday's papers and recent recommendations, ranks the remaining candidates, and uses Claude Code to produce detailed Chinese paper notes and a five-minute Chinese brief.

The workflow is assembled by daily_cookbook.yaml. Its schemas live in reme/schema/daily_paper.py, and its steps live in reme/steps/cookbook/daily_paper/.

Quick start

Daily Paper requires Python 3.11 or later, the core dependencies, network access to Hugging Face and arXiv, and credentials for the configured Claude Code endpoint.

From the repository root:

python -m pip install -e ".[dev,core]"
export CLAUDE_CODE_API_KEY="your-api-key"
reme start config=daily_cookbook job=daily_paper

The built-in configuration uses qwen3.7-max through DashScope's Anthropic-compatible endpoint. Override CLAUDE_CODE_MODEL_NAME and CLAUDE_CODE_BASE_URL when using another compatible model or provider.

By default, outputs are written under .reme/ in the directory where ReMe starts.

What it creates

A successful run writes ordinary PDFs and Markdown files beneath workspace_dir:

.reme/
├── daily/
│   ├── YYYY-MM-DD.md
│   └── YYYY-MM-DD/
│       ├── daily-paper-brief.md
│       ├── paper-<arxiv-id>.md
│       └── ...
├── resource/
│   └── papers/
│       ├── <arxiv-id>.pdf
│       └── ...
└── mem_session/
    └── claude_config/

• paper-<arxiv-id>.md is a detailed Chinese reading note with YAML frontmatter linking back to the source PDF and paper pages.
• daily-paper-brief.md is a roughly five-minute Chinese digest with wikilinks to every selected paper note.
• daily/YYYY-MM-DD.md is a derived day index rebuilt from the Markdown files for that date.
• resource/papers/ holds reusable source PDFs.

The paper notes are the source of truth for recommendation history: their frontmatter contains the arxiv_id values used for future deduplication. The day index is derived and can be rebuilt. The workflow does not currently write a separate run manifest.

How the workflow works

flowchart LR
    HF[Hugging Face<br/>weekly + monthly] --> C[1. Collect]
    Y[Yesterday's papers] --> C
    H[Recent local notes] --> C
    C --> R[2. Rank]
    R --> S[3. Select]
    S --> A[4. Analyze PDFs]
    A --> D[5. Build digest]
    D --> N[6. Notify DingTalk]
    A --> P[PDFs + paper notes]
    D --> B[Brief + day index]

1. Collect and deduplicate

The Collect step fetches the weekly ranking for the run date's ISO week, the monthly ranking for its calendar month, and the Hugging Face Daily Papers IDs for exactly the previous calendar day. It merges weekly and monthly metadata by arXiv ID and preserves each list's display rank.

It then scans daily/<prior-date>/paper-*.md over the configured history window and excludes IDs found in note frontmatter. The job fails clearly if no eligible papers remain.

2. Rank candidates

The Rank step uses reciprocal-rank fusion:

score = 1 / (rrf_k + monthly_rank)
      + weekly_weight / (rrf_k + weekly_rank)

A missing rank contributes zero. Candidates are ordered by fused score, upvotes, and arXiv ID. The bounded candidate pool also reserves several positions for papers whose titles or summaries match memory-related terms such as agent memory, memory retrieval, continual learning, context compression, knowledge graphs, and RAG. This reserve is a simple keyword heuristic, not a semantic classifier.

3. Select papers

Claude Code receives the bounded candidate pool and returns a structured PaperSelection. The implementation requires exactly top_k unique in-pool IDs with consecutive ranks. Invalid output is returned to the agent once as validation feedback; a second invalid response fails the job.

4. Download and analyze PDFs

Selected papers are processed sequentially. For each paper, the workflow:

1. validates the modern arXiv ID format;
2. downloads and validates the PDF, or reuses an existing file with a valid %PDF- header;
3. extracts text with pypdf, adding page markers and applying page and character limits;
4. asks Claude Code for a structured detailed reading; and
5. writes normalized frontmatter plus the generated Markdown body.

The current extractor requires a usable PDF text layer. Scanned or image-only PDFs fail because there is no OCR fallback. If extraction exceeds a configured limit, the note records that the input was truncated.

5. Build the brief and index

Claude Code reads every detailed note and produces the daily brief. The code verifies that each source-note wikilink is present and appends any missing links before writing the file. It then rebuilds daily/YYYY-MM-DD.md from that day's Markdown frontmatter.

6. Optionally notify DingTalk

The final step sends the brief body, without YAML frontmatter, to each configured DingTalk group in order. With no conversation IDs it is a no-op. If one group fails, the step still attempts the remaining groups and reports the combined failure afterward.

Dates, reruns, and idempotency

• date must be an exact YYYY-MM-DD value. When omitted, the job uses today in the application timezone, which is Asia/Shanghai in the built-in configuration.
• “Yesterday” means date - 1 day, not the previous 24 hours.
• history_days considers prior dated note directories only; the current run date is never part of its history scan.
• If daily/<date>/daily-paper-brief.md already exists and force=false, collection, ranking, model calls, PDF work, and digest generation are skipped. The existing brief remains available to the DingTalk notification step.
• force=true regenerates the notes and brief. Existing valid PDFs are still reused.

Each PDF, detailed note, and final brief uses a temporary file followed by replacement so callers do not see a partially written file. The complete multi-file workflow is not transactional, and there is no global lock for two concurrent runs of the same date.

Running the cookbook

The standalone configuration defines three jobs:

Job	Behavior
daily_paper	On-demand generation through the CLI or HTTP service
daily_paper_cron	The same pipeline every day at 08:00 in Asia/Shanghai
dingtalk_wait	A supervised background DingTalk agent

One-time runs

Generate today's brief:

reme start config=daily_cookbook job=daily_paper

Generate a specific date with selected overrides:

reme start \
  config=daily_cookbook \
  job=daily_paper \
  date=2026-07-21 \
  top_k=3 \
  history_days=30

Regenerate a date whose brief already exists:

reme start config=daily_cookbook job=daily_paper date=2026-07-21 force=true

Add service.show_metadata=true to a one-time command when the response metadata is useful for diagnostics.

Long-running service and cron

Start the standalone HTTP service and its scheduled/background jobs:

reme start config=daily_cookbook

It listens on 127.0.0.1:8001 by default, so it can run beside the default ReMe service. Call the on-demand job from another terminal with either the ReMe client or HTTP:

reme daily_paper host=127.0.0.1 port=8001

curl -s http://127.0.0.1:8001/daily_paper \
  -H 'Content-Type: application/json' \
  -d '{"date":"2026-07-21","top_k":3,"force":false}'

Service and schedule settings can be overridden at startup:

reme start \
  config=daily_cookbook \
  service.host=0.0.0.0 \
  service.port=8101 \
  jobs.daily_paper_cron.cron="30 7 * * *"

Configuration

The most useful job settings are:

Setting	Default	Purpose
candidate_limit	20	Maximum number of papers sent to selection
memory_reserve	5	Candidate positions reserved by the memory-keyword heuristic
top_k	3	Number of papers selected and analyzed
rrf_k	60	Reciprocal-rank fusion constant
weekly_weight	0.7	Weight of the weekly ranking in fusion
history_days	30	Prior recommendation window excluded by arXiv ID
hf_timeout	30 seconds	Hugging Face request timeout
hf_max_retries	3	Maximum Hugging Face request attempts
pdf_timeout	90 seconds	arXiv download timeout
max_pdf_bytes	52428800	Maximum PDF size (50 MiB)
max_pdf_pages	80	Maximum pages extracted for analysis
max_pdf_chars	240000	Maximum extracted characters sent for one paper

The public job parameters are date, force, top_k, weekly_weight, and history_days. Explicit invocation values take precedence over the job defaults.

The standalone application also accepts these environment variables:

Variable	Purpose
DAILY_PAPER_WORKSPACE_DIR	Overrides the default .reme workspace
DAILY_PAPER_PROJECT_PATH	Repository/project path visible to Claude Code
DAILY_PAPER_HOST / DAILY_PAPER_PORT	HTTP bind address
CLAUDE_CODE_API_KEY	API key for the configured Claude Code endpoint
CLAUDE_CODE_MODEL_NAME	Model override
CLAUDE_CODE_BASE_URL	Anthropic-compatible endpoint override

DAILY_PAPER_PROJECT_PATH defaults to .. relative to the workspace. With the default .reme workspace, starting from the repository root resolves it back to the repository. If the workspace lives elsewhere, set both paths explicitly.

ReMe loads an uncommitted .env file found from the current directory upward, so the same values may be placed there instead of exported in the shell.

DingTalk configuration

DingTalk is optional. Configure it only when brief delivery or the background DingTalk agent is needed:

DINGTALK_APP_KEY=your-app-key
DINGTALK_APP_SECRET=your-app-secret
DINGTALK_ROBOT_CODE=your-robot-code
DINGTALK_CONVERSATION_IDS=cid-group-one,cid-group-two

DINGTALK_CONVERSATION_IDS is required only for proactive brief delivery. The background dingtalk_wait job uses the first three credentials but not the conversation list.

Failure recovery and boundaries

Situation	Behavior
Temporary Hugging Face failure	Retries with exponential delay up to hf_max_retries attempts
No eligible papers	Fails before ranking
Invalid top_k or selection output	Fails after validation; selection output gets one retry
Oversized, invalid, or textless PDF	Stops during analysis
PDF exceeds page or character limits	Continues with truncated text and records the truncation
One paper analysis fails	Stops the job; earlier PDFs and notes remain on disk
Brief misses a source-note link	Appends the missing wikilink before writing
Existing final brief	Skips generation unless force=true; notification can still send it
Concurrent runs for one date	No pipeline-level lock; later writes may replace earlier results

To recover, inspect the date's notes and PDFs, fix the network, credential, model, or PDF issue, then rerun the same date with force=true. Valid cached PDFs will be reused.

The built-in Claude Code component runs with permission_mode: bypassPermissions. ReMe disables Claude Code's WebSearch tool, and the analysis/digest prompts constrain what the agent should read, but these steps do not set a strict per-call tool allowlist or an operating-system sandbox. Run the cookbook only with a trusted project and workspace, and tighten the agent configuration before shared or production use.

Tests

The focused unit suite mocks Hugging Face, arXiv, Claude Code, and DingTalk boundaries:

pytest tests/unit/test_daily_paper.py -v

Real runs access external services and may incur model costs; they should not be used as ordinary unit tests.