mirror of
https://github.com/showlab/Code2Video.git
synced 2026-08-28 06:54:58 +00:00
Update README.md
This commit is contained in:
Kevin
GitHub
parent
d9a4a69212
commit
bbd8de2a71
1 changed files with 589 additions and 230 deletions
819
README.md
819
README.md
|
|
@ -1,230 +1,589 @@
|
|||
|
||||
# Code2Video: Video Generation via Code
|
||||
|
||||
<!-- <p align="center">
|
||||
<img src="figures/logo.png" alt="Logo" width="30" style="vertical-align: middle; margin-right: 10px;"/>
|
||||
<span style="font-size: 1.8em; font-weight: bold;">Code2Video: Agentic Code-Centric Framework for Educational Video Generation</span>
|
||||
</p> -->
|
||||
|
||||
<!-- <p align="center">
|
||||
<img src="figures/logo.png" alt="Logo" width="30"/>
|
||||
</p> -->
|
||||
|
||||
<!-- <p align="center">
|
||||
<img src="figures/logo.png" alt="Logo" width="30" style="vertical-align: middle; margin-right: 10px;"/>
|
||||
<span style="font-size: 1.8em; font-weight: bold;"><em> From code to classroom-ready videos, powered by agents that teach.</em></span>
|
||||
</p> -->
|
||||
|
||||
<p align="center">
|
||||
<b>Code2Video: A Code-centric Paradigm for Educational Video Generation</b>
|
||||
</p>
|
||||
<video src="assets/video.mp4" width="600" controls>
|
||||
Your browser does not support the video tag.
|
||||
</video>
|
||||
|
||||
<!-- <p align="center">
|
||||
<em>From code to classroom-ready videos, powered by agents that teach.</em>
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<em>教学相长，代码为梁；知识作航，动画生光</em>
|
||||
</p> -->
|
||||
|
||||
|
||||
<p align="center">
|
||||
<a href="https://scholar.google.com.hk/citations?user=9lIMS-EAAAAJ&hl=zh-CN&oi=sra">Yanzhe Chen</a>,
|
||||
<a href="https://qhlin.me/">Kevin Qinghong Lin</a>,
|
||||
<a href="https://scholar.google.com/citations?user=h1-3lSoAAAAJ&hl=en">Mike Zheng Shou</a> <br>
|
||||
Show Lab @ National University of Singapore
|
||||
</p>
|
||||
|
||||
|
||||
<p align="center">
|
||||
<a href="https://arxiv.org/abs/2510.01174">📄 Paper</a> |
|
||||
<a href="https://huggingface.co/papers/2510.01174">🤗 Daily Paper</a> |
|
||||
<a href="https://huggingface.co/datasets/YanzheChen/MMMC">🤗 Dataset</a> |
|
||||
<a href="https://showlab.github.io/Code2Video/">🌐 Project Website</a> |
|
||||
<a href="https://x.com/KevinQHLin/status/1973673721824190864">💬 X (Twitter)</a>
|
||||
</p>
|
||||
|
||||
https://github.com/user-attachments/assets/d906423f-734a-41c9-b102-b113ad3b3c25
|
||||
|
||||
---
|
||||
|
||||
### Table of Contents
|
||||
- [🌟 Overview](#-overview)
|
||||
- [🚀 Quick Start: Code2Video](#-how-to-create----code2video)
|
||||
- [1. Requirements](#1-requirements)
|
||||
- [2. Configure LLM API Keys](#2-configure-llm-api-keys)
|
||||
- [3. Run Agents](#3-run-agents)
|
||||
- [4. Project Organization](#4-project-organization)
|
||||
- [📊 Evaluation: MMMC](#-how-to-evaluate----mmmc)
|
||||
- [🙏 Acknowledgements](#-acknowledgements)
|
||||
- [📌 Citation](#-citation)
|
||||
|
||||
---
|
||||
|
||||
## 🌟 Overview
|
||||
|
||||
<p align="center">
|
||||
<img src="figures/first.png" alt="Overview" width="90%">
|
||||
</p>
|
||||
|
||||
**Code2Video** is an **agentic, code-centric framework** that generates high-quality **educational videos** from knowledge points.
|
||||
Unlike pixel-based text-to-video models, our approach leverages executable **Manim code** to ensure **clarity, coherence, and reproducibility**.
|
||||
|
||||
**Key Features**:
|
||||
- 🎬 **Code-Centric Paradigm** — executable code as the unified medium for both temporal sequencing and spatial organization of educational videos.
|
||||
- 🤖 **Modular Tri-Agent Design** — Planner (storyboard expansion), Coder (debuggable code synthesis), and Critic (layout refinement with anchors) work together for structured generation.
|
||||
- 📚 **MMMC Benchmark** — the first benchmark for code-driven video generation, covering 117 curated learning topics inspired by 3Blue1Brown, spanning diverse areas.
|
||||
- 🧪 **Multi-Dimensional Evaluation** — systematic assessment on efficiency, aesthetics, and end-to-end knowledge transfer.
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Try Code2Video
|
||||
|
||||
<p align="center">
|
||||
<img src="figures/approach.png" alt="Approach" width="85%">
|
||||
</p>
|
||||
|
||||
### 1. Requirements
|
||||
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
````
|
||||
|
||||
### 2. Configure LLM API Keys
|
||||
|
||||
Fill in your **API credentials** in `gpt_config.json`.
|
||||
|
||||
* **LLM API**:
|
||||
* Required for Planner & Coder.
|
||||
* Best Manim code quality achieved with **Claude-4-Opus**.
|
||||
* **VLM API**:
|
||||
* Required for Planner Critic.
|
||||
* For layout and aesthetics optimization, provide **Gemini API key**.
|
||||
* Best quality achieved with **gemini-2.5-pro-preview-05-06**.
|
||||
|
||||
* **Visual Assets API**:
|
||||
|
||||
* To enrich videos with icons, set `ICONFINDER_API_KEY` from [IconFinder](https://www.iconfinder.com/account/applications).
|
||||
|
||||
### 3. Run Agents
|
||||
|
||||
We provide two shell scripts for different generation modes:
|
||||
|
||||
#### (a) Any Query
|
||||
|
||||
Script: `run_agent_single.sh`
|
||||
|
||||
Generates a video from a single **knowledge point** specified in the script.
|
||||
|
||||
```bash
|
||||
sh run_agent_single.sh --knowledge_point "Linear transformations and matrices"
|
||||
```
|
||||
|
||||
**Important parameters inside `run_agent_single.sh`:**
|
||||
|
||||
* `API`: specify which LLM to use.
|
||||
* `FOLDER_PREFIX`: output folder prefix (e.g., `TEST-single`).
|
||||
* `KNOWLEDGE_POINT`: target concept, e.g. `"Linear transformations and matrices"`.
|
||||
|
||||
---
|
||||
|
||||
#### (b) Full Benchmark Mode
|
||||
|
||||
Script: `run_agent.sh`
|
||||
|
||||
Runs all (or a subset of) learning topics defined in `long_video_topics_list.json`.
|
||||
|
||||
```bash
|
||||
sh run_agent.sh
|
||||
```
|
||||
|
||||
**Important parameters inside `run_agent.sh`:**
|
||||
|
||||
* `API`: specify which LLM to use.
|
||||
* `FOLDER_PREFIX`: name prefix for saving output folders (e.g., `TEST-LIST`).
|
||||
* `MAX_CONCEPTS`: number of concepts to include (`-1` means all).
|
||||
* `PARALLEL_GROUP_NUM`: number of groups to run in parallel.
|
||||
|
||||
### 4. Project Organization
|
||||
|
||||
A suggested directory structure:
|
||||
|
||||
```
|
||||
src/
|
||||
│── agent.py
|
||||
│── run_agent.sh
|
||||
│── run_agent_single.sh
|
||||
│── api_config.json
|
||||
│── ...
|
||||
│
|
||||
├── assets/
|
||||
│ ├── icons/ # downloaded visual assets cache via IconFinder API
|
||||
│ └── reference/ # reference images
|
||||
│
|
||||
├── json_files/ # JSON-based topic lists & metadata
|
||||
├── prompts/ # prompt templates for LLM calls
|
||||
├── CASES/ # generated cases, organized by FOLDER_PREFIX
|
||||
│ └── TEST-LIST/ # example multi-topic generation results
|
||||
│ └── TEST-single/ # example single-topic generation results
|
||||
```
|
||||
|
||||
|
||||
---
|
||||
|
||||
## 📊 Evaluation -- MMMC
|
||||
|
||||
We evaluate along **three complementary dimensions**:
|
||||
|
||||
1. **Knowledge Transfer (TeachQuiz)**
|
||||
|
||||
```bash
|
||||
python3 eval_TQ.py
|
||||
```
|
||||
|
||||
2. **Aesthetic & Structural Quality (AES)**
|
||||
|
||||
```bash
|
||||
python3 eval_AES.py
|
||||
```
|
||||
|
||||
3. **Efficiency Metrics (During Creating)**
|
||||
|
||||
* Token usage
|
||||
* Execution time
|
||||
|
||||
|
||||
👉 More data and evaluation scripts are available at:
|
||||
[HuggingFace: MMMC Benchmark](https://huggingface.co/datasets/YanzheChen/MMMC)
|
||||
|
||||
---
|
||||
|
||||
## 🙏 Acknowledgements
|
||||
|
||||
* Video data is sourced from the **[3Blue1Brown official lessons](https://www.3blue1brown.com/#lessons)**.
|
||||
These videos represent the **upper bound of clarity and aesthetics** in educational video design and inform our evaluation metrics.
|
||||
* We thank all the **Show Lab @ NUS** members for support!
|
||||
* This project builds upon open-source contributions from **Manim Community** and the broader AI research ecosystem.
|
||||
* High-quality visual assets (icons) are provided by **[IconFinder](https://www.iconfinder.com/)** and **[Icons8](https://icons8.com/icons)**, which were used to enrich the educational videos.
|
||||
|
||||
|
||||
---
|
||||
|
||||
## 📌 Citation
|
||||
|
||||
If you find our work useful, please cite:
|
||||
|
||||
```bibtex
|
||||
@misc{code2video,
|
||||
title={Code2Video: A Code-centric Paradigm for Educational Video Generation},
|
||||
author={Yanzhe Chen and Kevin Qinghong Lin and Mike Zheng Shou},
|
||||
year={2025},
|
||||
eprint={2510.01174},
|
||||
archivePrefix={arXiv},
|
||||
primaryClass={cs.CV},
|
||||
url={https://arxiv.org/abs/2510.01174},
|
||||
}
|
||||
```
|
||||
|
||||
<!DOCTYPE html>
|
||||
<!-- saved from url=(0025)https://qinghonglin.github.io/ -->
|
||||
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
|
||||
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
|
||||
<link rel="shortcut icon" href="https://qinghonglin.github.io/myIcon.ico">
|
||||
<meta name="google-site-verification" content="PcjE-PoDvp7KoKeZ5wE1g_BU8VI5wioTfiAgbIst__4" />
|
||||
<meta name="keywords" content="Qinghong Lin">
|
||||
<meta name="description" content="Qinghong Lin's homepage">
|
||||
<!-- <link rel="icon" href="icon.ico" type="figures/emoji"> -->
|
||||
<!-- <link rel="shortcut icon" href="https://em-content.zobj.net/thumbs/120/apple/325/technologist-light-skin-tone_1f9d1-1f3fb-200d-1f4bb.png"> -->
|
||||
<!-- <link rel="shortcut icon" href="https://em-content.zobj.net/thumbs/120/apple/325/snowflake_2744-fe0f.png"> -->
|
||||
<link rel="shortcut icon" href="./figures/world.png">
|
||||
<link rel="stylesheet" href="./index_files/jemdoc.css" type="text/css">
|
||||
<title>Qinghong Lin @ National University of Singapore</title>
|
||||
<script async="" src="./index_files/analytics.js"></script>
|
||||
<script type="text/javascript" async="" src="./index_files/ga.js"></script>
|
||||
<script type="text/javascript">
|
||||
var _gaq = _gaq || [];
|
||||
_gaq.push(['_setAccount', 'UA-39824124-1']);
|
||||
_gaq.push(['_trackPageview']);
|
||||
(function () {
|
||||
var ga = document.createElement('script');
|
||||
ga.type = 'text/javascript';
|
||||
ga.async = true;
|
||||
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
|
||||
var s = document.getElementsByTagName('script')[0];
|
||||
s.parentNode.insertBefore(ga, s);
|
||||
})();
|
||||
</script>
|
||||
</head>
|
||||
|
||||
<body>
|
||||
<div id="layout-content" style="margin-top:25px">
|
||||
<!-- <table> -->
|
||||
<table style="margin-bottom: -20px;">
|
||||
<tbody>
|
||||
<tr>
|
||||
<td width="670">
|
||||
<div id="toptitle">
|
||||
|
||||
<!-- <h1>Kevin Qinghong Lin</h1> -->
|
||||
<h1><span style="font-family: 'Comic Sans MS', Georgia; color: #6495ED;">Kevin</span>
|
||||
Qinghong Lin</h1>
|
||||
</div>
|
||||
<!-- <h3>Ph.D. Student</h3> -->
|
||||
<h3>Postdoctoral Researcher</h3>
|
||||
<p>
|
||||
<a href="https://torrvision.com/index.html">Torr Vision Group</a><br />
|
||||
<a href="https://www.ox.ac.uk/">University of Oxford</a><br />
|
||||
<br>
|
||||
Email: <u><a href="mailto:kevin.qh.lin@gmail.com">kevin.qh.lin [at] gmail.com</a></u>
|
||||
</p>
|
||||
<p>
|
||||
<a href="https://scholar.google.com/citations?user=EvbGjlUAAAAJ"><img
|
||||
src="./index_files/google_scholar.png" height="30px"></a>
|
||||
<a href="https://github.com/QinghongLin"><img src="./index_files/github.png"
|
||||
height="30px"></a>
|
||||
<!-- <a href="https://www.linkedin.com/in/lqh/"><img src="./index_files/linkedin.png"
|
||||
height="30px"></a> -->
|
||||
<a href="https://twitter.com/KevinQHLin"><img src="./index_files/x.png" height="30px"></a>
|
||||
</p>
|
||||
</td>
|
||||
<td>
|
||||
<img src="./figures/kevin_new.jpeg" border="0" width="250"><br>
|
||||
<!-- <img src="./figures/kevin.jpeg" border="0" width="250"><br> -->
|
||||
<!-- <p style="font-size: 10px; font-family: 'Comic Sans MS', Georgia; text-align: left;">Photo taken on <a href="https://en.wikipedia.org/wiki/Rottnest_Island">Rottnest Island</a>.</p> -->
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<h2>Biography</h2>
|
||||
<!-- <img src="./figures/wordcloud.png" style="float: right; width: 280px; margin: 0 0 10px 10px;"> -->
|
||||
<p>
|
||||
</p>
|
||||
<div style="text-align:justify">
|
||||
I am a Postdoctoral
|
||||
Researcher in <a href="https://torrvision.com/index.html">University of Oxford</a>, working with <a href="https://scholar.google.com/citations?user=kPxa2w0AAAAJ">Prof. Philip
|
||||
Torr</a>.
|
||||
<p></p>
|
||||
I successfully defended my PhD thesis in <a href="https://sites.google.com/view/showlab/home">National University of
|
||||
Singapore</a>, luckily advised
|
||||
by <a href="https://scholar.google.com/citations?user=h1-3lSoAAAAJ&hl">Prof. Mike
|
||||
Shou</a>.
|
||||
<p></p>
|
||||
<p></p>
|
||||
I was fortunate to intern at Tencent / Meta AI / Meta Reality Labs / Microsoft Research.
|
||||
<p></p>
|
||||
<p>I work on building multi-modal assistants <em>from</em> and <em>for</em> humans. This involves abilities
|
||||
like:</p>
|
||||
<ul>
|
||||
<li>
|
||||
Perception: video understanding (<a href="https://videomind.github.io/">VideoMind</a>, <a href="https://showlab.github.io/videollm-online/">VideoLLM-online</a>), video-language pretraining (<a href="https://arxiv.org/abs/2206.01670">EgoVLP</a>,
|
||||
<a href="https://arxiv.org/abs/2307.16715">UniVTG</a>)
|
||||
</li>
|
||||
<li>
|
||||
Reasoning:
|
||||
unified multimodal model (<a href="https://github.com/showlab/Show-o">Show-o</a>), reinforcement learning (<a href="https://arxiv.org/abs/2505.16854">Think or Not</a>, <a href="https://arxiv.org/abs/2508.08189">RL in Vision</a>)
|
||||
<li>
|
||||
Interaction: computer-use agents (<a href="https://github.com/showlab/ShowUI">ShowUI</a>, <a href="https://uivision.github.io/">UI-Vision</a>), vision agents (<a href="https://github.com/Paper2Poster/Paper2Poster">Paper2Poster</a>, <a href="https://showlab.github.io/Code2Video/">Code2Video</a>)
|
||||
</li>
|
||||
</ul>
|
||||
</ul>
|
||||
<p>I am looking for self-motivated students to work on the above topic. Feel free to drop me an email if you are interested.</p>
|
||||
<h2>News</h2>
|
||||
<div style="max-height: 300px; overflow-y: auto">
|
||||
<ul>
|
||||
<li>
|
||||
2025 Oct: Check out our newest work <a href="https://showlab.github.io/Code2Video/">Code2Video</a>: generate educational videos by code!
|
||||
</li>
|
||||
<li>
|
||||
2025 Sept: <a href="https://paper2poster.github.io/">Paper2Poster</a>, <a href="https://arxiv.org/abs/2505.16854">Think or Not</a> got accepted by
|
||||
<a href="https://neurips.cc/">NeurIPS
|
||||
2025</a>.
|
||||
</li>
|
||||
<li>
|
||||
2025 July: <a href="https://paper2poster.github.io/">Paper2Poster</a> is selected as an <a
|
||||
style="color:#FA8072">Oral</a> by
|
||||
<a href="https://icml.cc/virtual/2025/workshop/39955">ICML
|
||||
Multi-Agent Systems
|
||||
workshop
|
||||
2025</a>.
|
||||
</li>
|
||||
<li>
|
||||
2025 July: <a href="https://arxiv.org/abs/2406.13719">GUI-Narrator</a> got accepted by <a
|
||||
href="https://acmmm2025.org/">ACM MM
|
||||
2025</a>.
|
||||
</li>
|
||||
<li>
|
||||
2025 Jun: Selected for <a
|
||||
href="https://cvpr.thecvf.com/Conferences/2025/CallForDoctoralConsortium"
|
||||
style="color:#FA8072;">CVPR 2025 Doctoral Consortium</a>, Thank you!
|
||||
</li>
|
||||
<li>
|
||||
2025 May: <a href="https://arxiv.org/abs/2503.15661">UI-Vision</a> got accepted by <a
|
||||
href="https://icml.cc/">ICML 2025</a>.
|
||||
</li>
|
||||
<li>
|
||||
2025 Apr: Served as <a style="color:#FA8072">Area Chair</a> of <a
|
||||
href="https://neurips.cc/">NeurIPS
|
||||
2025</a>.
|
||||
</li>
|
||||
<li>
|
||||
2025 Feb: <a href="https://arxiv.org/abs/2411.17465">ShowUI</a>, <a
|
||||
href="http://arxiv.org/abs/2503.09402">VLog</a>, <a
|
||||
href="https://roictrl.github.io/">RoICtrl</a>, <a
|
||||
href="https://weijiawu.github.io/MovieBench/">MovieBench</a> got accepted by <a
|
||||
href="https://cvpr.thecvf.com/">CVPR 2025</a>.
|
||||
</li>
|
||||
<li>
|
||||
2025 Jan: <a href="https://showlab.github.io/Show-o/">Show-o</a> got accepted by <a
|
||||
href="https://www.iclr.cc/Conferences/2025">ICLR 2025</a>. Congrats to the team!
|
||||
</li>
|
||||
<li>
|
||||
2024 Dec: <a href="https://arxiv.org/abs/2411.17465">ShowUI</a> (<a
|
||||
style="color:#FA8072">Oral</a>)
|
||||
received <a style="color:#FA8072"><u>Outstanding Paper Award</u></a> by <a
|
||||
href="https://sites.google.com/view/open-world-agents/home">NeurIPS Open-World Agents
|
||||
workshop
|
||||
2024</a>.
|
||||
</li>
|
||||
<li>
|
||||
2024 Nov: Recognized as <a
|
||||
href="https://neurips.cc/Conferences/2024/ProgramCommittee#top-reviewers"
|
||||
style="color:#FA8072;">NeurIPS 2024 Top Reviewers</a>.
|
||||
</li>
|
||||
<li>
|
||||
2024 Sept: <a href="https://showlab.github.io/VideoGUI/">VideoGUI</a> (<a
|
||||
style="color:#FA8072">Spotlight</a>), <a
|
||||
href="https://arxiv.org/abs/2408.16730">VideoLLM-MoD</a> got accepted by <a
|
||||
href="https://neurips.cc/">NeurIPS 2024</a>.
|
||||
</li>
|
||||
<li>
|
||||
2024 Aug: <a href="https://dl.acm.org/doi/abs/10.1145/3688865.3689482">AssistGPT</a> got
|
||||
accepted by
|
||||
<a href="https://hcma2024.github.io/">HCMA@ACM MM 2024</a> as <a style="color:#FA8072"><u>Best
|
||||
Demo
|
||||
Paper</u></a>.
|
||||
</li>
|
||||
<li>
|
||||
2024 July: <a href="https://arxiv.org/abs/2407.21757">MovieSeq</a> got accepted by <a
|
||||
href="https://eccv.ecva.net/">ECCV 2024</a>.
|
||||
</li>
|
||||
<li>
|
||||
2024 Jun: <a href="https://arxiv.org/abs/2206.01670">EgoVLP</a> received <a
|
||||
href="https://egovis.github.io/awards/2022_2023/" style="color:#FA8072;"><u>Egocentric
|
||||
Vision
|
||||
(EgoVis) Distinguished Paper Award</u></a>.
|
||||
</li>
|
||||
<li>
|
||||
2024 May: Recognized as <a href="https://x.com/CVPR/status/1793616950314369239"
|
||||
style="color:#FA8072;">CVPR 2024 Outstanding Reviewers</a>.
|
||||
</li>
|
||||
<li>
|
||||
2024 Feb: <a
|
||||
href="https://openaccess.thecvf.com//content/CVPR2024/papers;Chen_VideoLLM-online_Online_Video_Large_Language_Model_for_Streaming_Video_CVPR_2024_paper.pdf">VideoLLM-online</a>,
|
||||
<a href="https://arxiv.org/abs/2312.01987">SparseFormer</a> got accepted by <a
|
||||
href="https://cvpr.thecvf.com/">CVPR 2024</a>.
|
||||
</li>
|
||||
<li>
|
||||
2023 Sept: <a href="https://arxiv.org/abs/2305.13777">VisorGPT</a> got accepted by <a
|
||||
href="https://nips.cc/">NeurIPS 2023</a>.
|
||||
</li>
|
||||
<li>
|
||||
2023 Aug: <a href="https://arxiv.org/abs/2206.01670">EgoVLP</a> received <a
|
||||
href="http://www.premiasg.org/" , style="color:#FA8072;"><u>PREMIA Best Student Paper Award
|
||||
(Gold award)</u></a>.
|
||||
</li>
|
||||
<li>
|
||||
2023 July: <a href="https://arxiv.org/abs/2307.16715">UniVTG</a>, <a
|
||||
href="https://arxiv.org/abs/2307.05463">EgoVLPv2</a>, <a
|
||||
href="https://arxiv.org/abs/2305.20087">TL;DR</a> got accepted by <a
|
||||
href="https://iccv2023.thecvf.com/">ICCV 2023</a>.
|
||||
</li>
|
||||
<li>
|
||||
2023 Mar: <a
|
||||
href="https://openaccess.thecvf.com/content/CVPR2023/html/Wang_All_in_One_Exploring_Unified_Video-Language_Pre-Training_CVPR_2023_paper.html">All-in-one</a>,
|
||||
<a
|
||||
href="https://openaccess.thecvf.com/content/CVPR2023/html/Chen_Affordance_Grounding_From_Demonstration_Video_To_Target_Image_CVPR_2023_paper.html">Afformer</a>
|
||||
got accepted by <a href="https://cvpr2023.thecvf.com/">CVPR 2023</a>.
|
||||
</li>
|
||||
<li>
|
||||
2022 Sept: <a href="https://arxiv.org/abs/2206.01670">EgoVLP</a> (<a
|
||||
style="color:#FA8072">Spotlight</a>) got accepted by <a href="https://nips.cc/">NeurIPS
|
||||
2022</a>.
|
||||
</li>
|
||||
<li>
|
||||
2022 Aug: Joined <a href="https://sites.google.com/view/showlab">Show Lab @ NUS</a> to start my
|
||||
Ph.D. journey!
|
||||
</li>
|
||||
<li>
|
||||
2022 Jun: <a href="https://arxiv.org/abs/2206.01670">EgoVLP</a> won <a
|
||||
style="color:#FA8072">Double
|
||||
Champions</a> of
|
||||
<a href="https://sites.google.com/view/cvpr2022w-ego4d-epic/">Joint 1st
|
||||
Ego4D and 10th EPIC Workshop, CVPR 2022</a>. <a
|
||||
href="https://cde.nus.edu.sg/ece/news/double-champions-at-the-2022-ieee-cvf-computer-vision-and-pattern-recognition-cvpr-epic-kitchens-challenges-and-ego4d-challenges/">[News]</a>
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
|
||||
<h2>Selected Publications <a href="https://scholar.google.com/citations?user=EvbGjlUAAAAJ">[Google
|
||||
Scholar]</a></h2>
|
||||
† indicates equal contribution. <u style="text-decoration-color: #6495ED;">Denotes student I mentored.</u>
|
||||
<table id="tbPublications" width="100%">
|
||||
<tbody>
|
||||
<tr>
|
||||
<td width="260">
|
||||
<img src="figures/code2video.png" width="240px" style="box-shadow: 4px 4px 8px #888">
|
||||
</td>
|
||||
<td><a href="https://arxiv.org/abs/2510.01174"><b>Code2Video: A Code-centric Paradigm for Educational Video Generation</b></a><br>
|
||||
<u style="text-decoration-color: #6495ED;">Yanzhe Chen†</u>, <u>Kevin QH. Lin†</u>, Mike Z. Shou.<br>
|
||||
<p style="margin-top:3px">
|
||||
<em>Preprint, </em>2025<br>
|
||||
[<a href="https://showlab.github.io/Code2Video/">project</a>]
|
||||
[<a href="https://arxiv.org/abs/2510.01174">paper</a>]
|
||||
[<a href="https://github.com/showlab/Code2Video">code</a>]
|
||||
[<a href="https://huggingface.co/datasets/YanzheChen/MMMC">dataset</a>]
|
||||
<br>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td width="260">
|
||||
<img src="figures/paper2poster.png" width="240px" style="box-shadow: 4px 4px 8px #888">
|
||||
</td>
|
||||
<td><a href="https://arxiv.org/abs/2505.21497"><b>Paper2Poster: Towards Multimodal Poster
|
||||
Automation from Scientific Papers</b></a><br>
|
||||
<u style="text-decoration-color: #6495ED;">Wei Pang†</u>, <u>Kevin QH. Lin†</u>, Xiangru
|
||||
Jian†, Xi He, Philip Torr<br>
|
||||
<p style="margin-top:3px">
|
||||
<em>NeurIPS D&B, </em>2025<br>
|
||||
<em>ICML MAS workshop, </em>2025. <a style="color:#FA8072">Oral</a><br>
|
||||
[<a href="https://arxiv.org/abs/2505.21497">paper</a>]
|
||||
[<a href="https://github.com/Paper2Poster/Paper2Poster">code</a>]
|
||||
[<a href="https://paper2poster.github.io/">project</a>]
|
||||
[<a href="https://huggingface.co/datasets/Paper2Poster/Paper2Poster">datasets</a>]
|
||||
[<a href="https://x.com/_akhaliq/status/1927721150584390129">twitter</a>]
|
||||
<br>
|
||||
<a style="color:#FA8072">2.5K github stars.</a>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td width="260">
|
||||
<img src="figures/ton.png" width="240px" style="box-shadow: 4px 4px 8px #888">
|
||||
</td>
|
||||
<td><a href="https://arxiv.org/abs/2505.16854"><b>Think or Not? Selective Reasoning via
|
||||
Reinforcement Learning for Vision-Language Models</b></a><br>
|
||||
<u style="text-decoration-color: #6495ED;">Jiaqi Wang†</u>, <u>Kevin QH. Lin†</u>, James
|
||||
Cheng, Mike Z. Shou.<br>
|
||||
<p style="margin-top:3px">
|
||||
<em>NeurIPS, </em>2025<br>
|
||||
[<a href="https://arxiv.org/abs/2505.16854">paper</a>]
|
||||
[<a href="https://github.com/kokolerk/TON">code</a>]
|
||||
[<a
|
||||
href="https://huggingface.co/collections/kolerk/ton-682ad9038395c21e228a645b">huggingface</a>]
|
||||
<br>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td width="260">
|
||||
<img src="figures/videomind.jpg" width="240px" style="box-shadow: 4px 4px 8px #888">
|
||||
</td>
|
||||
<td><a href="https://arxiv.org/abs/2503.13444"><b>VideoMind: A Chain-of-LoRA Agent for Long
|
||||
Video Reasoning</b></a><br>
|
||||
Ye Liu†, <u>Kevin QH. Lin†</u>, Chang Wen Chen, Mike Z. Shou.<br>
|
||||
<p style="margin-top:3px">
|
||||
<em>Preprint, </em>2025<br>
|
||||
[<a href="https://arxiv.org/abs/2503.13444">paper</a>]
|
||||
[<a href="https://github.com/yeliudev/VideoMind">code</a>]
|
||||
[<a href="https://huggingface.co/datasets/yeliudev/VideoMind-Dataset">dataset</a>]
|
||||
[<a href="https://videomind.github.io/">project</a>]
|
||||
[<a href="https://huggingface.co/spaces/yeliudev/VideoMind-2B">demo</a>]
|
||||
<br>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td width="260">
|
||||
<img src="figures/vla.png" width="240px" style="box-shadow: 4px 4px 8px #888">
|
||||
</td>
|
||||
<td><a href="https://huggingface.co/papers/2411.17465"><b>ShowUI: One Vision-Language-Action
|
||||
Model for GUI Visual Agent</b></a><br>
|
||||
<u>Kevin QH. Lin</u>, Linjie Li, Difei Gao, Zhengyuan Yang, Shiwei Wu, Zechen Bai, Stan WX.
|
||||
Lei, Lijuan Wang, Mike Z. Shou.<br>
|
||||
<p style="margin-top:3px">
|
||||
<em>CVPR, </em>2025<br>
|
||||
<em>NeurIPS OWA workshop, </em>2024. <a style="color:#FA8072">Oral</a><br>
|
||||
[<a href="https://arxiv.org/abs/2411.17465">paper</a>]
|
||||
[<a href="https://github.com/showlab/ShowUI">code</a>]
|
||||
[<a href="https://huggingface.co/showlab/ShowUI-2B">huggingface</a>]
|
||||
[<a href="https://huggingface.co/datasets/showlab/ShowUI-desktop-8K">dataset</a>]
|
||||
[<a href="https://huggingface.co/spaces/showlab/ShowUI">demo</a>]
|
||||
<br>
|
||||
<a style="color:#FA8072">Outstanding Paper Award, NeurIPS Open-World Agents Workshop
|
||||
2024.</a><br>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td width="260">
|
||||
<img src="figures/vlog.jpg" width="240px" style="box-shadow: 4px 4px 8px #888">
|
||||
</td>
|
||||
<td><a href="http://arxiv.org/abs/2503.09402"><b>VLog: Video-Language Models by Generative
|
||||
Retrieval of Narration Vocabulary</b></a><br>
|
||||
<u>Kevin QH. Lin</u>, Mike Z. Shou.<br>
|
||||
<p style="margin-top:3px">
|
||||
<!-- <em>NeurIPS OWA workshop, </em>2024. <a style="color:#FA8072">Oral</a><br> -->
|
||||
<em>CVPR, </em>2025<br>
|
||||
[<a href="http://arxiv.org/abs/2503.09402">paper</a>]
|
||||
[<a href="https://github.com/showlab/VLog">code</a>]
|
||||
<br>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td width="260">
|
||||
<img src="figures/videogui.jpg" width="240px" style="box-shadow: 4px 4px 8px #888">
|
||||
</td>
|
||||
<td><a href="https://showlab.github.io/VideoGUI/"><b>VideoGUI: A Benchmark for GUI Automation
|
||||
from Instructional Videos</b></a><br>
|
||||
<u>Kevin QH. Lin</u>, Linjie Li, Difei Gao, Qinchen Wu, Mingyi Yan, Zhengyuan Yang, Lijuan
|
||||
Wang, Mike Z. Shou.<br>
|
||||
<p style="margin-top:3px">
|
||||
<em>NeurIPS D&B, </em>2024. <a style="color:#FA8072">Spotlight</a><br>
|
||||
[<a href="https://showlab.github.io/VideoGUI/assets/preprint.pdf">paper</a>]
|
||||
[<a href="https://github.com/showlab/VideoGUI">code</a>]
|
||||
[<a href="https://showlab.github.io/VideoGUI/">project</a>]
|
||||
<br>
|
||||
<!-- <a style="color:#FA8072">Can an agent recreate PowerPoint animation effects from instructional videos?</a><br> -->
|
||||
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td width="260">
|
||||
<img src="figures/movieseq.png" width="240px" style="box-shadow: 4px 4px 8px #888">
|
||||
</td>
|
||||
<td><a href="https://arxiv.org/abs/2407.21757"><b>Learning Video Context as Interleaved
|
||||
Multimodal Sequences </b></a> <br><u>Kevin QH. Lin</u>, Pengchuan Zhang, Difei Gao,
|
||||
Xide Xia, Joya Chen, Ziteng Gao, Jinheng Xie, Xuhong Xiao, Mike Z. Shou.<br>
|
||||
<p style="margin-top:3px">
|
||||
<em>ECCV, </em>2024<br>
|
||||
[<a href="https://arxiv.org/abs/2407.21757">paper</a>]
|
||||
[<a href="https://github.com/showlab/MovieSeq">code</a>]
|
||||
<br>
|
||||
<!-- <a style="color:#FA8072">Video in-context learning using interleaved sequences of images, videos, plots and dialogues.</a><br> -->
|
||||
<br>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td width="260">
|
||||
<img src="figures/univtg.jpg" width="240px" height="120px"
|
||||
style="box-shadow: 4px 4px 8px #888">
|
||||
</td>
|
||||
<td><a href="https://arxiv.org/abs/2307.16715"><b>UniVTG: Towards Unified Video-Language
|
||||
Temporal Grounding</b></a> <br>
|
||||
<u>Kevin QH. Lin</u>, Pengchuan Zhang, Joya Chen, Shraman Pramanick, Difei Gao, Alex JP.
|
||||
Wang, Rui Yan, Mike Z. Shou.<br>
|
||||
<p style="margin-top:3px">
|
||||
<!-- <em>International Conference on Computer Vision (<b>ICCV</b>), </em>2023.<br> -->
|
||||
<em>ICCV, </em>2023<br>
|
||||
[<a href="https://arxiv.org/pdf/2307.16715.pdf">paper</a>]
|
||||
[<a href="https://github.com/showlab/UniVTG">code</a>]
|
||||
[<a href="https://huggingface.co/spaces/KevinQHLin/UniVTG">demo</a>]
|
||||
<br>
|
||||
<!-- <font color="#FF0000">The first video temporal grounding pretraining model, unifying diverse temporal annotations to power moment retrieval, highlight detection and video summarization.</font> -->
|
||||
<!-- <a style="color:#FA8072">The first video temporal grounding pretraining model, unifying diverse temporal labels.</a><br> -->
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td width="260">
|
||||
<img src="EgoVLP/static/images/framework.jpeg" width="240px" height="120px"
|
||||
style="box-shadow: 4px 4px 8px #888">
|
||||
</td>
|
||||
<!-- <td><a href="https://arxiv.org/abs/2206.01670"><b>Egocentric Video-Language Pretraining</b></a> -->
|
||||
<td><a href="https://arxiv.org/abs/2206.01670"><b>Egocentric Video-Language Pretraining</b></a>
|
||||
<br>
|
||||
<u>Kevin QH. Lin</u>, Alex JP. Wang, M. Soldan, M. Wray, R. Yan, Eric ZC. Xu, D. Gao, R. Tu,
|
||||
W. Zhao, W. Kong, C. Cai, H. Wang, D. Damen, B. Ghanem, W. Liu, Mike Z. Shou.<br>
|
||||
<p style="margin-top:3px">
|
||||
<!-- <em>Neural Information Processing Systems (<b>NeurIPS</b>), </em>2022. <font
|
||||
color="#FF0000"><b>Spotlight (1.7%)</b></font><br> -->
|
||||
<em>NeurIPS, </em>2022. <a style="color:#FA8072">Spotlight (1.7%)</a><br>
|
||||
[<a href="https://arxiv.org/pdf/2206.01670.pdf">paper</a>]
|
||||
[<a href="https://github.com/showlab/EgoVLP">code</a>]
|
||||
[<a href="https://qinghonglin.github.io/EgoVLP/">project</a>]
|
||||
[<a href="EgoVLP/static/images/poster.pdf">poster</a>]
|
||||
[<a
|
||||
href="https://cde.nus.edu.sg/ece/news-detail/double-champions-at-the-2022-ieee-cvf-computer-vision-and-pattern-recognition-cvpr-epic-kitchens-challenges-and-ego4d-challenges/">media</a>]
|
||||
<br>
|
||||
<!-- <font color="#FF0000">The first egocentric vision-language pretrained model.<br> -->
|
||||
<!-- <a style="color:#FA8072">The first egocentric vision-language pretrained model. </a><br> -->
|
||||
<a style="color:#FA8072">EgoVis Distinguished Paper Award & PREMIA Best Student Paper
|
||||
Award 2023.</a><br>
|
||||
<a style="color:#FA8072">Double champions in Ego4D & Epic-Kitchens CVPR 2022
|
||||
challenges.</a><br>
|
||||
<!-- Double champions in Ego4D & Epic-Kitchens CVPR 2022 challenges.</font> <a href=https://cde.nus.edu.sg/ece/news-detail/double-champions-at-the-2022-ieee-cvf-computer-vision-and-pattern-recognition-cvpr-epic-kitchens-challenges-and-ego4d-challenges/>[News]</a> -->
|
||||
</td>
|
||||
</tr>
|
||||
<tr></tr>
|
||||
<tr></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</ul>
|
||||
|
||||
<h2>Honors</h2>
|
||||
<ul>
|
||||
<li>
|
||||
<div style="float:left; text-align:left">CVPR Doctoral Consortium</div>
|
||||
<div style="float:right; text-align:right">2025</div>
|
||||
</li>
|
||||
<li>
|
||||
<div style="float:left; text-align:left">Outstanding Paper Award, NeurIPS Open-World Agents</div>
|
||||
<div style="float:right; text-align:right">2024</div>
|
||||
</li>
|
||||
<li>
|
||||
<div style="float:left; text-align:left">NeurIPS Top Reviewers</div>
|
||||
<div style="float:right; text-align:right">2024</div>
|
||||
</li>
|
||||
<li>
|
||||
<div style="float:left; text-align:left">Best Demo Paper Award, ACM Multimedia HCMA</div>
|
||||
<div style="float:right; text-align:right">2024</div>
|
||||
</li>
|
||||
<li>
|
||||
<div style="float:left; text-align:left">Egocentric Vision (EgoVis) Distinguished Paper Award</div>
|
||||
<div style="float:right; text-align:right">2024</div>
|
||||
</li>
|
||||
<li>
|
||||
<div style="float:left; text-align:left">CVPR Outstanding Reviewers (Top 2%)</div>
|
||||
<div style="float:right; text-align:right">2024</div>
|
||||
</li>
|
||||
<li>
|
||||
<div style="float:left; text-align:left">PREMIA Best Student Paper Awards, Gold Award</div>
|
||||
<div style="float:right; text-align:right">2023</div>
|
||||
</li>
|
||||
<li>
|
||||
<div style="float:left; text-align:left">NeurIPS Scholar Award</div>
|
||||
<div style="float:right; text-align:right">2022</div>
|
||||
</li>
|
||||
<li>
|
||||
<!-- <div style="float:left; text-align:left">Tencent Rhino-Bird Research Scholarship <a href=https://mp.weixin.qq.com/s/06e7m8twMen9DzrAE0raPA>[News]</a></div> -->
|
||||
<div style="float:left; text-align:left">Tencent Rhino-Bird Research Scholarship, Second Prize</a>
|
||||
</div>
|
||||
<div style="float:right; text-align:right">2022</div>
|
||||
</li>
|
||||
<li>
|
||||
<div style="float:left; text-align:left">1st Place on Ego4D - Object State Change Classification
|
||||
Challenge, CVPR</div>
|
||||
<div style="float:right; text-align:right">2022</div>
|
||||
</li>
|
||||
<li>
|
||||
<div style="float:left; text-align:left"></div>1st Place on EPIC-Kitchens - Multi-Instance Retrieval
|
||||
Challenge, CVPR<div style="float:right; text-align:right">2022</div>
|
||||
</li>
|
||||
<!--
|
||||
<li>
|
||||
<div style="float:left; text-align:left">2nd Place on Ego4D - Natural Language Queries Challenge, CVPR</div> <div style="float:right; text-align:right">2022</div>
|
||||
</li>
|
||||
-->
|
||||
<!-- <li>
|
||||
<div style="float:left; text-align:left">AAAI Student Scholarship</div>
|
||||
<div style="float:right; text-align:right">2022</div>
|
||||
</li>
|
||||
<li>
|
||||
<div style="float:left; text-align:left">SIGIR Student Travel Grant</div>
|
||||
<div style="float:right; text-align:right">2021</div>
|
||||
</li> -->
|
||||
<!-- <li>
|
||||
<div style="float:left; text-align:left">Outstanding Graduate at SZU</div> <div style="float:right; text-align:right">2022, 2019</div>
|
||||
</li>
|
||||
-->
|
||||
<li>
|
||||
<div style="float:left; text-align:left">Show Lab Annual Award</div>
|
||||
<div style="float:right; text-align:right">2022, 2024</div>
|
||||
</li>
|
||||
<li>
|
||||
<div style="float:left; text-align:left">China National Scholarship</div>
|
||||
<div style="float:right; text-align:right">2018, 2021</div>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
|
||||
<h2>Service</h2>
|
||||
<ul>
|
||||
<li>
|
||||
<p>Area Chair: NeurIPS 2025.
|
||||
</li>
|
||||
<li>
|
||||
<p>Workshop Organizer: <a href="https://showlab.github.io/omg/">Open Multimodal Gathering @ NUS</a>;
|
||||
<a href="https://sites.google.com/view/loveucvpr25/home">Multimodal Video Agent @ CVPR 25</a>.
|
||||
</li>
|
||||
<li>
|
||||
<!-- SDM -->
|
||||
<p>Conference Reviewer: CVPR (2024 Outstanding Reviewers), ICCV, ECCV, NeurIPS (2024 Top Reviewers),
|
||||
ICML, ICLR, etc.</p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Journal Reviewer: TPAMI, IJCV, TMLR, TNNLS, TMM, etc.</p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Teaching Assistant:
|
||||
<!-- <a href="https://nusmods.com/courses/EE6934/deep-learning-advanced">EE6934 Deep Learning</a>,
|
||||
<a href="https://nusmods.com/courses/EE6733/advanced-topics-on-vision-and-machine-learning">EE6733 Advanced Topics on Vision and Machine Learning</a>,
|
||||
<a href="https://nusmods.com/courses/EE6733/advanced-topics-on-vision-and-machine-learning">EE4212 Computer Vision</a> -->
|
||||
<a href="https://nusmods.com/courses/EE6934/deep-learning-advanced">EE6934</a>,
|
||||
<a
|
||||
href="https://nusmods.com/courses/EE6733/advanced-topics-on-vision-and-machine-learning">EE6733</a>,
|
||||
<a
|
||||
href="https://nusmods.com/courses/EE6733/advanced-topics-on-vision-and-machine-learning">EE4212</a>
|
||||
</p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Co-organizer of <a href="https://theaitalks.org/">The AI Talks.</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
</br>
|
||||
<div align="center">
|
||||
<!-- <a href="https://info.flagcounter.com/yZuS"><img src="https://s05.flagcounter.com/count2/yZuS/bg_FFFFFF/txt_000000/border_CCCCCC/columns_2/maxflags_10/viewers_0/labels_0/pageviews_1/flags_0/percent_0/" alt="Flag Counter" border="0"></a></div> -->
|
||||
<!-- <a href="https://info.flagcounter.com/yZuS"><img src="https://s05.flagcounter.com/map/yZuS/size_s/txt_000000/border_CCCCCC/pageviews_1/viewers_0/flags_0/" alt="Flag Counter" border="0"></a></div> -->
|
||||
<a href="https://info.flagcounter.com/yZuS"><img
|
||||
src="https://s05.flagcounter.com/mini/yZuS/bg_FFFFFF/txt_000000/border_CCCCCC/flags_0/"
|
||||
alt="Flag Counter" border="0"></a>
|
||||
</div>
|
||||
</br>
|
||||
|
||||
<div align="center">
|
||||
|
||||
<body>
|
||||
<font color="gray">© Kevin</font>
|
||||
</body>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
(function (i, s, o, g, r, a, m) {
|
||||
i['GoogleAnalyticsObject'] = r; i[r] = i[r] || function () {
|
||||
(i[r].q = i[r].q || []).push(arguments)
|
||||
}, i[r].l = 1 * new Date(); a = s.createElement(o),
|
||||
m = s.getElementsByTagName(o)[0]; a.async = 1; a.src = g; m.parentNode.insertBefore(a, m)
|
||||
})(window, document, 'script', 'https://www.google-analytics.com/analytics.js', 'ga');
|
||||
|
||||
ga('create', 'UA-88615920-1', 'auto');
|
||||
ga('send', 'pageview');
|
||||
</script>
|
||||
</div>
|
||||
</div>
|
||||
</body>
|
||||
|
||||
</html>
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue