Spaces:
Runtime error
Runtime error
File size: 20,030 Bytes
51ac68d d9486d1 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 |
---
title: TheoremExplainAgent
emoji: π
colorFrom: blue
colorTo: purple
sdk: gradio
sdk_version: "4.44.0"
app_file: app.py
pinned: false
license: mit
---
# π Theorem-Explain-Agent Video Generation
Generate educational videos explaining mathematical theorems and concepts using AI.
## Features
- π€ Multiple AI model support (Gemini, OpenAI, Anthropic)
- π¬ Automated video generation with Manim
- π΅ Text-to-speech narration
- π Real-time status tracking
- π Background processing
## How to Use
1. **Start Generation**: Enter a Model, Topic, and Context
2. **Get Task ID**: Copy the generated task ID
3. **Check Status**: Monitor progress in the Check Status tab
4. **Download**: Get your video when generation completes
## Supported Models
- `gemini/gemini-1.5-flash` (recommended)
- `gemini/gemini-1.5-pro`
- `openai/gpt-4o`
- `openai/o3-mini`
- `anthropic/claude-3-opus-20240229`
## Requirements
This application requires API keys for the AI models you want to use. Set them in the Space's Repository Secrets:
- `GEMINI_API_KEY`
- `OPENAI_API_KEY`
- `LANGFUSE_PUBLIC_KEY`
- `LANGFUSE_SECRET_KEY`
## Example Usage
**Topic**: "The Pythagorean Theorem"
**Context**: "In a right-angled triangle, the square of the length of the hypotenuse is equal to the sum of the squares of the other two sides."
**Model**: "gemini/gemini-1.5-flash"
The system will generate an educational video explaining the theorem with visual animations and narration.
# TheoremExplainAgent (TEA) π΅
[](https://arxiv.org/abs/2502.19400)
<a href='https://huggingface.co/papers/2502.19400'><img src='https://img.shields.io/static/v1?label=Paper&message=Huggingface&color=orange'></a>
[**π Homepage**](https://tiger-ai-lab.github.io/TheoremExplainAgent/) | [**π arXiv**](https://arxiv.org/abs/2502.19400) | [**π€ HuggingFace Dataset**](https://huggingface.co/datasets/TIGER-Lab/TheoremExplainBench) | [π₯Video Data](https://drive.google.com/file/d/18kmzXvbxaFGyJw-g51jnq9m93v_ez4aJ/view)
[](https://github.com/TIGER-AI-Lab/TheoremExplainAgent/graphs/contributors)
[](https://github.com/TIGER-AI-Lab/TheoremExplainAgent/blob/main/LICENSE)
[](https://github.com/TIGER-AI-Lab/TheoremExplainAgent)
[](https://hits.seeyoufarm.com)
This repo contains the codebase for our paper [TheoremExplainAgent: Towards Video-based Multimodal Explanations for LLM Theorem Understanding](https://arxiv.org/abs/2502.19400)
**ACL 2025 main**
## Introduction
TheoremExplainAgent is an AI system that generates long-form Manim videos to visually explain theorems, proving its deep understanding while uncovering reasoning flaws that text alone often hides.
https://github.com/user-attachments/assets/17f2f4f2-8f2c-4abc-b377-ac92ebda69f3
## π° News
* 2025 Jun 8: We released our generated video data for researchers to serve as baselines.
* 2025 May 15: Paper accepted to ACL 2025 main conference.
* 2025 Mar 3: Generation code and Evaluation code released. Thanks for the wait!
<!--* 2025 Mar 3: Reach 404 stars without code.-->
* 2025 Feb 27: Paper available on [Arxiv](https://arxiv.org/abs/2502.19400). Thanks AK for putting our paper on [HF Daily](https://huggingface.co/papers/2502.19400).
## Downloading Generated Video Data
Skip this section if you just want to try out the code.
If you are researchers who just need the baseline videos as baseline comparison, download it here:
```shell
wget --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=18kmzXvbxaFGyJw-g51jnq9m93v_ez4aJ' -O /tmp/gdrive.html && wget --load-cookies /tmp/cookies.txt -O baseline_videos.zip "https://drive.usercontent.google.com/download?id=18kmzXvbxaFGyJw-g51jnq9m93v_ez4aJ&export=download&confirm=$(sed -rn 's/.*name="confirm" value="([^"]+)".*/\\1/p' /tmp/gdrive.html)&uuid=$(sed -rn 's/.*name="uuid" value="([^"]+)".*/\\1/p' /tmp/gdrive.html)" && rm /tmp/gdrive.html /tmp/cookies.txt
```
## Installation
> **Look at the [FAQ section in this README doc](https://github.com/TIGER-AI-Lab/TheoremExplainAgent?tab=readme-ov-file#-faq) if you encountered any errors. If that didnt help, create a issue**<br>
1. Setting up conda environment
```shell
conda create --name tea python=3.12.8
conda activate tea
pip install -r requirements.txt
```
2. You may also need to install latex and other dependencies for Manim Community. Look at [Manim Installation Docs](https://docs.manim.community/en/stable/installation.html) for more details.
```shell
# You might need these dependencies if you are using Linux Ubuntu:
sudo apt-get install portaudio19-dev
sudo apt-get install libsdl-pango-dev
```
3. Then Download the Kokoro model and voices using the commands to enable TTS service.
```shell
mkdir -p models && wget -P models https://github.com/thewh1teagle/kokoro-onnx/releases/download/model-files/kokoro-v0_19.onnx && wget -P models https://github.com/thewh1teagle/kokoro-onnx/releases/download/model-files/voices.bin
```
4. Create `.env` based on `.env.template`, filling in the environmental variables according to the models you choose to use.
See [LiteLLM](https://docs.litellm.ai/docs/providers) for reference.
```shell
touch .env
```
Then open the `.env` file and edit it with whatever text editor you like.
Your `.env` file should look like the following:
```shell
# OpenAI
OPENAI_API_KEY=""
# Azure OpenAI
AZURE_API_KEY=""
AZURE_API_BASE=""
AZURE_API_VERSION=""
# Google Vertex AI
VERTEXAI_PROJECT=""
VERTEXAI_LOCATION=""
GOOGLE_APPLICATION_CREDENTIALS=""
# Google Gemini
GEMINI_API_KEY=""
...
# Kokoro TTS Settings
KOKORO_MODEL_PATH="models/kokoro-v0_19.onnx"
KOKORO_VOICES_PATH="models/voices.bin"
KOKORO_DEFAULT_VOICE="af"
KOKORO_DEFAULT_SPEED="1.0"
KOKORO_DEFAULT_LANG="en-us"
```
Fill in the API keys according to the model you wanted to use.
5. Configure Python path. Note that you need to configure the python path to make it work. Otherwise you may encounter import issues (like not being able to import src etc.)
```shell
export PYTHONPATH=$(pwd):$PYTHONPATH
```
6. (Optional) To setup RAG, See [https://github.com/TIGER-AI-Lab/TheoremExplainAgent?tab=readme-ov-file#generation-with-rag](https://github.com/TIGER-AI-Lab/TheoremExplainAgent?tab=readme-ov-file#generation-with-rag).
> **Look at the [FAQ section in this README doc](https://github.com/TIGER-AI-Lab/TheoremExplainAgent?tab=readme-ov-file#-faq) if you encountered any errors. If that didnt help, create a issue**<br>
## Generation
### Supported Models
<!--You can customize the allowed models by editing the `src/utils/allowed_models.json` file. This file specifies which `model` and `helper_model` the system is permitted to use.-->
The model naming follows the LiteLLM convention. For details on how models should be named, please refer to the [LiteLLM documentation](https://docs.litellm.ai/docs/providers).
### Generation (Single topic)
```shell
python generate_video.py \
--model "openai/o3-mini" \
--helper_model "openai/o3-mini" \
--output_dir "output/your_exp_name" \
--topic "your_topic" \
--context "description of your topic, e.g. 'This is a topic about the properties of a triangle'" \
```
Example:
```shell
python generate_video.py \
--model "openai/o3-mini" \
--helper_model "openai/o3-mini" \
--output_dir "output/my_exp_name" \
--topic "Big O notation" \
--context "most common type of asymptotic notation in computer science used to measure worst case complexity" \
```
### Generation (in batch)
```shell
python generate_video.py \
--model "openai/o3-mini" \
--helper_model "openai/o3-mini" \
--output_dir "output/my_exp_name" \
--theorems_path data/thb_easy/math.json \
--max_scene_concurrency 7 \
--max_topic_concurrency 20 \
```
### Generation with RAG
Before using RAG, download the RAG documentation from this [Google Drive link](https://drive.google.com/file/d/1Tn6J_JKVefFZRgZbjns93KLBtI9ullRv/view?usp=sharing). After downloading, unzip the file. For example, if you unzip it to `data/rag/manim_docs`, then you should set `--manim_docs_path` to `data/rag/manim_docs`. The vector database will be created the first time you run with RAG.
```shell
python generate_video.py \
--model "openai/o3-mini" \
--helper_model "openai/o3-mini" \
--output_dir "output/with_rag/o3-mini/vtutorbench_easy/math" \
--topic "Big O notation" \
--context "most common type of asymptotic notation in computer science used to measure worst case complexity" \
--use_rag \
--chroma_db_path "data/rag/chroma_db" \
--manim_docs_path "data/rag/manim_docs" \
--embedding_model "vertex_ai/text-embedding-005"
```
We support more options for generation, see below for more details:
```shell
usage: generate_video.py [-h]
[--model]
[--topic TOPIC] [--context CONTEXT]
[--helper_model]
[--only_gen_vid] [--only_combine] [--peek_existing_videos] [--output_dir OUTPUT_DIR] [--theorems_path THEOREMS_PATH]
[--sample_size SAMPLE_SIZE] [--verbose] [--max_retries MAX_RETRIES] [--use_rag] [--use_visual_fix_code]
[--chroma_db_path CHROMA_DB_PATH] [--manim_docs_path MANIM_DOCS_PATH]
[--embedding_model {azure/text-embedding-3-large,vertex_ai/text-embedding-005}] [--use_context_learning]
[--context_learning_path CONTEXT_LEARNING_PATH] [--use_langfuse] [--max_scene_concurrency MAX_SCENE_CONCURRENCY]
[--max_topic_concurrency MAX_TOPIC_CONCURRENCY] [--debug_combine_topic DEBUG_COMBINE_TOPIC] [--only_plan] [--check_status]
[--only_render] [--scenes SCENES [SCENES ...]]
Generate Manim videos using AI
options:
-h, --help show this help message and exit
--model Select the AI model to use
--topic TOPIC Topic to generate videos for
--context CONTEXT Context of the topic
--helper_model Select the helper model to use
--only_gen_vid Only generate videos to existing plans
--only_combine Only combine videos
--peek_existing_videos, --peek
Peek at existing videos
--output_dir OUTPUT_DIR
Output directory
--theorems_path THEOREMS_PATH
Path to theorems json file
--sample_size SAMPLE_SIZE, --sample SAMPLE_SIZE
Number of theorems to sample
--verbose Print verbose output
--max_retries MAX_RETRIES
Maximum number of retries for code generation
--use_rag, --rag Use Retrieval Augmented Generation
--use_visual_fix_code, --visual_fix_code
Use VLM to fix code with rendered visuals
--chroma_db_path CHROMA_DB_PATH
Path to Chroma DB
--manim_docs_path MANIM_DOCS_PATH
Path to manim docs
--embedding_model {azure/text-embedding-3-large,vertex_ai/text-embedding-005}
Select the embedding model to use
--use_context_learning
Use context learning with example Manim code
--context_learning_path CONTEXT_LEARNING_PATH
Path to context learning examples
--use_langfuse Enable Langfuse logging
--max_scene_concurrency MAX_SCENE_CONCURRENCY
Maximum number of scenes to process concurrently
--max_topic_concurrency MAX_TOPIC_CONCURRENCY
Maximum number of topics to process concurrently
--debug_combine_topic DEBUG_COMBINE_TOPIC
Debug combine videos
--only_plan Only generate scene outline and implementation plans
--check_status Check planning and code status for all theorems
--only_render Only render scenes without combining videos
--scenes SCENES [SCENES ...]
Specific scenes to process (if theorems_path is provided)
```
## Evaluation
Note that Gemini and GPT4o is required for evaluation.
Currently, evaluation requires a video file and a subtitle file (SRT format).
Video evaluation:
```shell
usage: evaluate.py [-h]
[--model_text {gemini/gemini-1.5-pro-002,gemini/gemini-1.5-flash-002,gemini/gemini-2.0-flash-001,vertex_ai/gemini-1.5-flash-002,vertex_ai/gemini-1.5-pro-002,vertex_ai/gemini-2.0-flash-001,openai/o3-mini,gpt-4o,azure/gpt-4o,azure/gpt-4o-mini,bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0,bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0,bedrock/anthropic.claude-3-5-haiku-20241022-v1:0,bedrock/us.anthropic.claude-3-7-sonnet-20250219-v1:0}]
[--model_video {gemini/gemini-1.5-pro-002,gemini/gemini-2.0-flash-exp,gemini/gemini-2.0-pro-exp-02-05}]
[--model_image {gemini/gemini-1.5-pro-002,gemini/gemini-1.5-flash-002,gemini/gemini-2.0-flash-001,vertex_ai/gemini-1.5-flash-002,vertex_ai/gemini-1.5-pro-002,vertex_ai/gemini-2.0-flash-001,openai/o3-mini,gpt-4o,azure/gpt-4o,azure/gpt-4o-mini,bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0,bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0,bedrock/anthropic.claude-3-5-haiku-20241022-v1:0,bedrock/us.anthropic.claude-3-7-sonnet-20250219-v1:0}]
[--eval_type {text,video,image,all}] --file_path FILE_PATH --output_folder OUTPUT_FOLDER [--retry_limit RETRY_LIMIT] [--combine] [--bulk_evaluate] [--target_fps TARGET_FPS]
[--use_parent_folder_as_topic] [--max_workers MAX_WORKERS]
Automatic evaluation of theorem explanation videos with LLMs
options:
-h, --help show this help message and exit
--model_text {gemini/gemini-1.5-pro-002,gemini/gemini-1.5-flash-002,gemini/gemini-2.0-flash-001,vertex_ai/gemini-1.5-flash-002,vertex_ai/gemini-1.5-pro-002,vertex_ai/gemini-2.0-flash-001,openai/o3-mini,gpt-4o,azure/gpt-4o,azure/gpt-4o-mini,bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0,bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0,bedrock/anthropic.claude-3-5-haiku-20241022-v1:0,bedrock/us.anthropic.claude-3-7-sonnet-20250219-v1:0}
Select the AI model to use for text evaluation
--model_video {gemini/gemini-1.5-pro-002,gemini/gemini-2.0-flash-exp,gemini/gemini-2.0-pro-exp-02-05}
Select the AI model to use for video evaluation
--model_image {gemini/gemini-1.5-pro-002,gemini/gemini-1.5-flash-002,gemini/gemini-2.0-flash-001,vertex_ai/gemini-1.5-flash-002,vertex_ai/gemini-1.5-pro-002,vertex_ai/gemini-2.0-flash-001,openai/o3-mini,gpt-4o,azure/gpt-4o,azure/gpt-4o-mini,bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0,bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0,bedrock/anthropic.claude-3-5-haiku-20241022-v1:0,bedrock/us.anthropic.claude-3-7-sonnet-20250219-v1:0}
Select the AI model to use for image evaluation
--eval_type {text,video,image,all}
Type of evaluation to perform
--file_path FILE_PATH
Path to a file or a theorem folder
--output_folder OUTPUT_FOLDER
Directory to store the evaluation files
--retry_limit RETRY_LIMIT
Number of retry attempts for each inference
--combine Combine all results into a single JSON file
--bulk_evaluate Evaluate a folder of theorems together
--target_fps TARGET_FPS
Target FPS for video processing. If not set, original video FPS will be used
--use_parent_folder_as_topic
Use parent folder name as topic name for single file evaluation
--max_workers MAX_WORKERS
Maximum number of concurrent workers for parallel processing
```
* For `file_path`, it is recommended to pass a folder containing both an MP4 file and an SRT file.
## Misc: Modify the system prompt in TheoremExplainAgent
If you want to modify the system prompt, you need to:
1. Modify files in `task_generator/prompts_raw` folder.
2. Run `task_generator/parse_prompt.py` to rebuild the `__init__.py` file.
```python
cd task_generator
python parse_prompt.py
cd ..
```
## TheoremExplainBench (TEB)
TheoremExplainBench can be found on https://huggingface.co/datasets/TIGER-Lab/TheoremExplainBench.
How to use:
```python
import datasets
dataset = datasets.load_dataset("TIGER-Lab/TheoremExplainBench")
```
Dataset info:
```shell
DatasetDict({
train: Dataset({
features: ['uid', 'subject', 'difficulty', 'theorem', 'description', 'subfield'],
num_rows: 240
})
})
```
## β FAQ
The FAQ should cover the most common errors you could encounter. If you see something new, report it on issues.
Q: Error `src.utils.kokoro_voiceover import KokoroService # You MUST import like this as this is our custom voiceover service. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ModuleNotFoundError: No module named 'src'`. <br>
A: Please run `export PYTHONPATH=$(pwd):$PYTHONPATH` when you start a new terminal. <br>
Q: Error `Files not found` <br>
A: Check your Manim installation. <br>
Q: Error `latex ...` <br>
A: Check your latex installation. <br>
Q: The output log is not showing response? <br>
A: It could be API-related issues. Make sure your `.env` file is properly configured (fill in your API keys), or you can enable litellm debug mode to figure out the issues. <be>
Q: Plans / Scenes are missing? <br>
A: It could be API-related issues. Make sure your `.env` file is properly configured (fill in your API keys), or you can enable litellm debug mode to figure out the issues. <br>
## ποΈ Citation
Please kindly cite our paper if you use our code, data, models or results:
```bibtex
@misc{ku2025theoremexplainagentmultimodalexplanationsllm,
title={TheoremExplainAgent: Towards Multimodal Explanations for LLM Theorem Understanding},
author={Max Ku and Thomas Chong and Jonathan Leung and Krish Shah and Alvin Yu and Wenhu Chen},
year={2025},
eprint={2502.19400},
archivePrefix={arXiv},
primaryClass={cs.AI},
url={https://arxiv.org/abs/2502.19400},
}
```
## π« License
This project is released under the [the MIT License](LICENSE).
## β Star History
[](https://star-history.com/#TIGER-AI-Lab/TheoremExplainAgent&Date)
## π Acknowledgements
We want to thank [Votee AI](https://votee.ai/) for sponsoring API keys to access the close-sourced models.
The code is built upon the below repositories, we thank all the contributors for open-sourcing.
* [Manim Community](https://www.manim.community/)
* [kokoro-manim-voiceover](https://github.com/xposed73/kokoro-manim-voiceover)
* [manim-physics](https://github.com/Matheart/manim-physics)
* [manim-Chemistry](https://github.com/UnMolDeQuimica/manim-Chemistry)
* [ManimML](https://github.com/helblazer811/ManimML)
* [manim-dsa](https://github.com/F4bbi/manim-dsa)
* [manim-circuit](https://github.com/Mr-FuzzyPenguin/manim-circuit)
## π¨ Disclaimer
**This work is intended for research purposes only. The authors do not encourage or endorse the use of this codebase for commercial applications. The code is provided "as is" without any warranties, and users assume all responsibility for its use.**
Tested Environment: MacOS, Linux
|