diff --git a/README.md b/README.md index 0081146..b625c09 100644 --- a/README.md +++ b/README.md @@ -1,230 +1,589 @@ - -# Code2Video: Video Generation via Code - - - - - - - -

- Code2Video: A Code-centric Paradigm for Educational Video Generation -

- - - - - -

- Yanzhe Chen, - Kevin Qinghong Lin, - Mike Zheng Shou
- Show Lab @ National University of Singapore -

- - -

-  📄 Paper   |   -  🤗 Daily Paper   |   -  🤗 Dataset   |   -  🌐 Project Website   |   -  💬 X (Twitter) -

- -https://github.com/user-attachments/assets/d906423f-734a-41c9-b102-b113ad3b3c25 - ---- - -### Table of Contents -- [🌟 Overview](#-overview) -- [🚀 Quick Start: Code2Video](#-how-to-create----code2video) - - [1. Requirements](#1-requirements) - - [2. Configure LLM API Keys](#2-configure-llm-api-keys) - - [3. Run Agents](#3-run-agents) - - [4. Project Organization](#4-project-organization) -- [📊 Evaluation: MMMC](#-how-to-evaluate----mmmc) -- [🙏 Acknowledgements](#-acknowledgements) -- [📌 Citation](#-citation) - ---- - -## 🌟 Overview - -

- Overview -

- -**Code2Video** is an **agentic, code-centric framework** that generates high-quality **educational videos** from knowledge points. -Unlike pixel-based text-to-video models, our approach leverages executable **Manim code** to ensure **clarity, coherence, and reproducibility**. - -**Key Features**: -- 🎬 **Code-Centric Paradigm** — executable code as the unified medium for both temporal sequencing and spatial organization of educational videos. -- 🤖 **Modular Tri-Agent Design** — Planner (storyboard expansion), Coder (debuggable code synthesis), and Critic (layout refinement with anchors) work together for structured generation. -- 📚 **MMMC Benchmark** — the first benchmark for code-driven video generation, covering 117 curated learning topics inspired by 3Blue1Brown, spanning diverse areas. -- 🧪 **Multi-Dimensional Evaluation** — systematic assessment on efficiency, aesthetics, and end-to-end knowledge transfer. - ---- - -## 🚀 Try Code2Video - -

- Approach -

- -### 1. Requirements - -```bash -pip install -r requirements.txt -```` - -### 2. Configure LLM API Keys - -Fill in your **API credentials** in `gpt_config.json`. - -* **LLM API**: - * Required for Planner & Coder. - * Best Manim code quality achieved with **Claude-4-Opus**. -* **VLM API**: - * Required for Planner Critic. - * For layout and aesthetics optimization, provide **Gemini API key**. - * Best quality achieved with **gemini-2.5-pro-preview-05-06**. - -* **Visual Assets API**: - - * To enrich videos with icons, set `ICONFINDER_API_KEY` from [IconFinder](https://www.iconfinder.com/account/applications). - -### 3. Run Agents - -We provide two shell scripts for different generation modes: - -#### (a) Any Query - -Script: `run_agent_single.sh` - -Generates a video from a single **knowledge point** specified in the script. - -```bash -sh run_agent_single.sh --knowledge_point "Linear transformations and matrices" -``` - -**Important parameters inside `run_agent_single.sh`:** - -* `API`: specify which LLM to use. -* `FOLDER_PREFIX`: output folder prefix (e.g., `TEST-single`). -* `KNOWLEDGE_POINT`: target concept, e.g. `"Linear transformations and matrices"`. - ---- - -#### (b) Full Benchmark Mode - -Script: `run_agent.sh` - -Runs all (or a subset of) learning topics defined in `long_video_topics_list.json`. - -```bash -sh run_agent.sh -``` - -**Important parameters inside `run_agent.sh`:** - -* `API`: specify which LLM to use. -* `FOLDER_PREFIX`: name prefix for saving output folders (e.g., `TEST-LIST`). -* `MAX_CONCEPTS`: number of concepts to include (`-1` means all). -* `PARALLEL_GROUP_NUM`: number of groups to run in parallel. - -### 4. Project Organization - -A suggested directory structure: - -``` -src/ -│── agent.py -│── run_agent.sh -│── run_agent_single.sh -│── api_config.json -│── ... -│ -├── assets/ -│ ├── icons/ # downloaded visual assets cache via IconFinder API -│ └── reference/ # reference images -│ -├── json_files/ # JSON-based topic lists & metadata -├── prompts/ # prompt templates for LLM calls -├── CASES/ # generated cases, organized by FOLDER_PREFIX -│ └── TEST-LIST/ # example multi-topic generation results -│ └── TEST-single/ # example single-topic generation results -``` - - ---- - -## 📊 Evaluation -- MMMC - -We evaluate along **three complementary dimensions**: - -1. **Knowledge Transfer (TeachQuiz)** - - ```bash - python3 eval_TQ.py - ``` - -2. **Aesthetic & Structural Quality (AES)** - - ```bash - python3 eval_AES.py - ``` - -3. **Efficiency Metrics (During Creating)** - - * Token usage - * Execution time - - -👉 More data and evaluation scripts are available at: -[HuggingFace: MMMC Benchmark](https://huggingface.co/datasets/YanzheChen/MMMC) - ---- - -## 🙏 Acknowledgements - -* Video data is sourced from the **[3Blue1Brown official lessons](https://www.3blue1brown.com/#lessons)**. - These videos represent the **upper bound of clarity and aesthetics** in educational video design and inform our evaluation metrics. -* We thank all the **Show Lab @ NUS** members for support! -* This project builds upon open-source contributions from **Manim Community** and the broader AI research ecosystem. -* High-quality visual assets (icons) are provided by **[IconFinder](https://www.iconfinder.com/)** and **[Icons8](https://icons8.com/icons)**, which were used to enrich the educational videos. - - ---- - -## 📌 Citation - -If you find our work useful, please cite: - -```bibtex -@misc{code2video, - title={Code2Video: A Code-centric Paradigm for Educational Video Generation}, - author={Yanzhe Chen and Kevin Qinghong Lin and Mike Zheng Shou}, - year={2025}, - eprint={2510.01174}, - archivePrefix={arXiv}, - primaryClass={cs.CV}, - url={https://arxiv.org/abs/2510.01174}, -} -``` - + + + + + + + + + + + + + + + + Qinghong Lin @ National University of Singapore + + + + + + +
+ + + + + + + + + + +
+
+ + +

Kevin + Qinghong Lin

+
+ +

Postdoctoral Researcher

+

+ Torr Vision Group
+ University of Oxford
+
+ Email: kevin.qh.lin [at] gmail.com +

+

+ + + + +

+ 		+
+ + +
+ +

Biography

+ +

+

+
+ I am a Postdoctoral + Researcher in University of Oxford, working with Prof. Philip + Torr. +

+ I successfully defended my PhD thesis in National University of + Singapore, luckily advised + by Prof. Mike + Shou. +

+

+ I was fortunate to intern at Tencent / Meta AI / Meta Reality Labs / Microsoft Research. +

+

I work on building multi-modal assistants from and for humans. This involves abilities + like:

+ + +

I am looking for self-motivated students to work on the above topic. Feel free to drop me an email if you are interested.

+

News

+
+ +
+ +

Selected Publications [Google + Scholar]

+ † indicates equal contribution. Denotes student I mentored. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + Code2Video: A Code-centric Paradigm for Educational Video Generation
+ Yanzhe Chen†, Kevin QH. Lin†, Mike Z. Shou.
+

+ Preprint, 2025
+ [project] + [paper] + [code] + [dataset] +
+

+ + Paper2Poster: Towards Multimodal Poster + Automation from Scientific Papers
+ Wei Pang†, Kevin QH. Lin†, Xiangru + Jian†, Xi He, Philip Torr
+

+ NeurIPS D&B, 2025
+ ICML MAS workshop, 2025. Oral
+ [paper] + [code] + [project] + [datasets] + [twitter] +
+ 2.5K github stars. +

+ + Think or Not? Selective Reasoning via + Reinforcement Learning for Vision-Language Models
+ Jiaqi Wang†, Kevin QH. Lin†, James + Cheng, Mike Z. Shou.
+

+ NeurIPS, 2025
+ [paper] + [code] + [huggingface] +
+

+ + VideoMind: A Chain-of-LoRA Agent for Long + Video Reasoning
+ Ye Liu†, Kevin QH. Lin†, Chang Wen Chen, Mike Z. Shou.
+

+ Preprint, 2025
+ [paper] + [code] + [dataset] + [project] + [demo] +
+

+ + ShowUI: One Vision-Language-Action + Model for GUI Visual Agent
+ Kevin QH. Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Shiwei Wu, Zechen Bai, Stan WX. + Lei, Lijuan Wang, Mike Z. Shou.
+

+ CVPR, 2025
+ NeurIPS OWA workshop, 2024. Oral
+ [paper] + [code] + [huggingface] + [dataset] + [demo] +
+ Outstanding Paper Award, NeurIPS Open-World Agents Workshop + 2024.
+

+ + VLog: Video-Language Models by Generative + Retrieval of Narration Vocabulary
+ Kevin QH. Lin, Mike Z. Shou.
+

+ + CVPR, 2025
+ [paper] + [code] +
+

+ + VideoGUI: A Benchmark for GUI Automation + from Instructional Videos
+ Kevin QH. Lin, Linjie Li, Difei Gao, Qinchen Wu, Mingyi Yan, Zhengyuan Yang, Lijuan + Wang, Mike Z. Shou.
+

+ NeurIPS D&B, 2024. Spotlight
+ [paper] + [code] + [project] +
+ + +

+ + Learning Video Context as Interleaved + Multimodal Sequences
Kevin QH. Lin, Pengchuan Zhang, Difei Gao, + Xide Xia, Joya Chen, Ziteng Gao, Jinheng Xie, Xuhong Xiao, Mike Z. Shou.
+

+ ECCV, 2024
+ [paper] + [code] +
+ +
+

+ + UniVTG: Towards Unified Video-Language + Temporal Grounding
+ Kevin QH. Lin, Pengchuan Zhang, Joya Chen, Shraman Pramanick, Difei Gao, Alex JP. + Wang, Rui Yan, Mike Z. Shou.
+

+ + ICCV, 2023
+ [paper] + [code] + [demo] +
+ + +

+ + Egocentric Video-Language Pretraining +
+ Kevin QH. Lin, Alex JP. Wang, M. Soldan, M. Wray, R. Yan, Eric ZC. Xu, D. Gao, R. Tu, + W. Zhao, W. Kong, C. Cai, H. Wang, D. Damen, B. Ghanem, W. Liu, Mike Z. Shou.
+

+ + NeurIPS, 2022. Spotlight (1.7%)
+ [paper] + [code] + [project] + [poster] + [media] +
+ + + EgoVis Distinguished Paper Award & PREMIA Best Student Paper + Award 2023.
+ Double champions in Ego4D & Epic-Kitchens CVPR 2022 + challenges.
+ +

+ + +

Honors

+ + + +

Service

+ + +
+
+ + + Flag Counter +
+
+ +
+ + + © Kevin + +
+ + +
+
+ + +