CE-Reporter: Model Weights Collection
Overview
This is a Hugging Face Collection for CE-Reporter, an innovative AI-driven system designed for automated capsule endoscopy (CE) video-to-report generation. Our work, detailed in the paper "Towards Expert-Level Generative AI for Capsule Endoscopy Video-to-Report Generation" (under review), addresses the challenge of generating concise, evidence-supported diagnostic reports from lengthy CE videos. CE-Reporter leverages vision-language integration to achieve high-quality report generation with improved efficiency and transparency in gastrointestinal diagnostics.
This collection contains the model weights for the three core components of CE-Reporter:
- CE-Reporter-kfd: Keyframe detection model for identifying critical frames in CE videos.
- CE-Reporter-llava-1.5-7b: Vision-language model for generating captions from keyframes.
- CE-Reporter-llama-3.1-8b: Language model for aggregating captions and generating comprehensive diagnostic reports.
Paper
For more details about the methodology, dataset, and performance, refer to our paper:
"Towards Expert-Level Generative AI for Capsule Endoscopy Video-to-Report Generation" (under review).
Authors: Boyun Zheng, Yu Jiang, Dejun Fan, Xinyu Liu, Xiaoqing Guo, Shengyuan Liu, Bin Chen, Mingjie Wang, Jinni Luo, Jin Tao, Jiancong Hu, Lichao Sun, Lei Xing, Yixuan Yuan.
Models
This collection includes the following model weights:
| Model | Description | Hugging Face Repository |
|---|---|---|
| ce-reporter-kfd | Keyframe detection model for identifying diagnostically relevant frames in CE videos. | Byzzz0301/CE-Reporter-kfd |
| ce-reporter-llava-1.5-7b | Vision-language model based on LLaVA for generating descriptive captions from keyframes. | Byzzz0301/CE-Reporter-llava-1.5-7b |
| ce-reporter-llama-3.1-8b | Language model based on Llama for generating concise and accurate diagnostic reports. | Byzzz0301/CE-Reporter-llama-3.1-8b |
Usage
To use these models, please refer to the individual repository READMEs for detailed instructions on setup, inference, and training. For more information on the CE-Reporter system, including environment setup and data preparation, visit the CE-Reporter GitHub repository.
Contact
For questions or data access requests, please contact the corresponding authors: Yixuan Yuan. Data access requests will be evaluated per institutional policies and ethical standards.