datasets:
- Homie0609/MatchTime
language:
- en
license: cc-by-sa-4.0
tags:
- sports
- soccer
pipeline_tag: video-text-to-text
library_name: transformers
Commentary Generation for Soccer Highlights
This repository contains the code and model for Commentary Generation for Soccer Highlights, as presented in our paper:
Commentary Generation for Soccer Highlights
Abstract
Automated soccer commentary generation has evolved from template-based systems to advanced neural architectures, aiming to produce real-time descriptions of sports events. While frameworks like SoccerNet-Caption laid foundational work, their inability to achieve fine-grained alignment between video content and commentary remains a significant challenge. Recent efforts such as MatchTime, with its MatchVoice model, address this issue through coarse and fine-grained alignment techniques, achieving improved temporal synchronization. In this paper, we extend MatchVoice to commentary generation for soccer highlights using the GOAL dataset, which emphasizes short clips over entire games. We conduct extensive experiments to reproduce the original MatchTime results and evaluate our setup, highlighting the impact of different training configurations and hardware limitations. Furthermore, we explore the effect of varying window sizes on zero-shot performance. While MatchVoice exhibits promising generalization capabilities, our findings suggest the need for integrating techniques from broader video-language domains to further enhance performance.
\u25b6\ufe0fDemo Video (YouTube) \u25b6\ufe0fDemo Video (bilibili) \u00b7 \ud83c\udfe0Project Page \u00b7 \ud83d\udcbbCode \u00b7 \ud83d\udcddOriginal Paper (MatchTime) \u00b7 \ud83d\udccaDataset \u00b7 \ud83d\udce5Checkpoint
Requirements
- Python >= 3.8 (Recommend to use Anaconda or Miniconda)
- PyTorch >= 2.0.0 (If use A100)
- transformers >= 4.42.3
- pycocoevalcap >= 1.2
A suitable conda environment named matchtime can be created and activated with:
cd MatchTime
conda env create -f environment.yaml
conda activate matchtime
Training
Before training, make sure you have prepared features and caption data, and put them into according folders. The structure after collating should be like:
└─ MatchTime
├─ dataset
│ ├─ MatchTime
│ │ ├─ valid
│ │ └─ train
│ │ ├─ england_epl_2014-2015
│ │ ... ├─ 2015-02-21 - 18-00 Chelsea 1 - 1 Burnley
│ │ ... └─ Labels-caption.json
│ │
│ ├─ SN-Caption
│ └─ SN-Caption-test-align
│ ├─ england_epl_2015-2016
│ ... ├─ 2015-08-16 - 18-00 Manchester City 3 - 0 Chelsea
│ ... └─ Labels-caption_with_gt.json
│
├─ features
│ ├─ baidu_soccer_embeddings
│ │ ├─ england_epl_2014-2015
... │ ... ├─ 2015-02-21 - 18-00 Chelsea 1 - 1 Burnley
│ ... ├─ 1_baidu_soccer_embeddings.npy
│ └─ 2_baidu_soccer_embeddings.npy
├─ C3D_PCA512
...
with the format of features is adjusted by
python ./features/preprocess.py directory_path_of_feature
Above example gives the format of Baidu feature, in our experiments we also used ResNET_PCA_512, C3D_PCA_512 from official website. If you want to use CLIP(2 FPS) or InternVideo(1FPS) feature. You can follow their official website to extract feature or contact us for features.
After preparing the data and features, you can pre-train (or finetune) with the following terminal command (Check hyper-parameters at the bottom of train.py):
python train.py
Inference
We provide two types of inference:
For all test set
You can generate a .csv file with the following code to test the MatchVoice model with the following code (Check hyper-parameters at the bottom of inference.py)
python inference.py
There is a sample of this type of inference in ./inference_result/sample.csv.
For Single Video
We also provide a version for predict the commentary single video (for our checkpoints, use 30s video)
python inference_single_video_CLIP.py single_video_path
Here we only provide the version of CLIP feature (using VIT/B-32), for crop the CLIP feature, please check here. CLIP features are not the one with best performance but are the most friendly for new new videos.
Alignment
Before doing alignment, you should download videos from here (224p is enough) and make it in the following format:
└─ MatchTime
├─ videos_224p
... ├─ england_epl_2014-2015
... ├─ 2015-02-21 - 18-00 Chelsea 1 - 1 Burnley
... ├─ 1_224.mkv
└─ 2_224p.mkv
Pre-process (Coarse Align)
We need to use WhisperX and LLaMA3(as agent) to finish coarse alignment with following steps:
WhisperX ASR:
python ./alignment/soccer_whisperx.py --process_directory video_folder(eg. ./videos_224p/england_epl_2014-2015) --output_directory output_folder(eg. ./ASR_results/england_epl_2014-2015)
Transform to Events:
python ./alignment/soccer_asr2events.py --base_path ASR_results_folder(eg. ./ASR_results/england_epl_2014-2015) --output_dir envent_results_folder(eg. ./event_results/england_epl_2014-2015)
Align from Events:
python ./alignment/soccer_align_from_event.py --event_path envent_results_folder(eg. ./event_results/england_epl_2014-2015) --output_dir output_directory(eg. ./pre-processed/england_epl_2014-2015)
More details could be checked in paper.
Contrastive Learning (Fine-grained Align)
After downloading checkpoints from here. Use the following code to finish alignment with contrastive learning:
python ./alignment/do_alignment.py
By changing the hyper-parameter finding_words, you can freely align from ASR, enent, or original SN-Caption.
Also, you can directly use alignment model by
from alignment.matchtime_model import ContrastiveLearningModel
Evaluation
We provide codes for evaluate the prediction results:
# for single csv file
python ./evaluation/scoer_single.py --csv_path ./inference_result/sample.csv
# for many csv files to record scores in a new csv file
python ./evaluation/scoer_group.py
# for gpt score (need OpenAI API Key)
python ./evaluation/scoer_gpt.py ./inference_result/sample.csv
Citation
If you use this code for your research or project, please cite:
@article{rao2024matchtimeautomaticsoccergame,
title={MatchTime: Towards Automatic Soccer Game Commentary Generation},
author={Jiayuan Rao and Haoning Wu and Chang Liu and Yanfeng Wang and Weidi Xie},
year={2024},
journal={arXiv preprint arXiv:2406.18530},
}
@article{rao2024commentary,
title={Commentary Generation for Soccer Highlights},
author={Rao, Jiayuan and Wu, Haoning and Liu, Chang and Wang, Yanfeng and Xie, Weidi},
journal={arXiv preprint arXiv:2508.07543},
year={2024},
}
Acknowledgements
Many thanks to the code bases from Video-LLaMA and source data from SoccerNet-Caption.
Contact
If you have any questions, please feel free to contact jy_rao@sjtu.edu.cn or haoningwu3639@gmail.com.