| # Evaluation | |
| ## Batch Evaluation | |
| To evaluate the model on a dataset, use the `batch_eval.py` script. It is significantly more efficient in large-scale evaluation compared to `demo.py`, supporting batched inference, multi-GPU inference, torch compilation, and skipping video compositions. | |
| An example of running this script with four GPUs is as follows: | |
| ```bash | |
| OMP_NUM_THREADS=4 torchrun --standalone --nproc_per_node=4 batch_eval.py duration_s=8 dataset=vggsound model=small_16k num_workers=8 | |
| ``` | |
| You may need to update the data paths in `config/eval_data/base.yaml`. | |
| More configuration options can be found in `config/base_config.yaml` and `config/eval_config.yaml`. | |
| You might also want to change the dataset definition if you are not evaluating on VGGSound: https://github.com/hkchengrex/MMAudio/blob/main/mmaudio/data/eval/video_dataset.py | |
| ## Precomputed Results | |
| Precomputed results for VGGSound, AudioCaps, and MovieGen are available here: https://huggingface.co/datasets/hkchengrex/MMAudio-precomputed-results | |
| ## Obtaining Quantitative Metrics | |
| Our evaluation code is available here: https://github.com/hkchengrex/av-benchmark | |