This repository contains the code for the paper “C$^2$-Cite: Contextual-Aware Citation Generation for \\ Attributed Large Language Models”. The project is based on the open-source repository"[TUDB-Labs/MoE-PEFT](https://github.com/TUDB-Labs/MoE-PEFT)". C$^2$-Cite is a model that can answer the questions with citation markers.
## File description
- **config**: Including the configurations of training or evaluating
- **c2cite/backends**: Some backend tools for GMoE.
- **c2cite/common**: The implementation of Transformer architecture.
- **c2cite/models**: The implementation of some series of Transformer-based models.
- **c2cite/tasks**: The implementation of datasets.
- **c2cite.py** The start file of this project.
## Environment Requirements
 - python3=3.11
 - pytorch >= 2.1.2
 - Other dependencies, See ```requirements.txt```
## Quick Start
### STEP 1: Download Base models
 - [Llama-3-8B-inst]
### STEP 2: Downlaod training datasets
To get Training dataset proposed in paper "Towards Faithful and Robust LLM Specialists for Evidence-Based Question-Answering", you can download [SynSciQA](https://github.com/EdisonNi-hku/Robust_Evidence_Based_QA) here. And please put SynSciQA.json, SynSciQA+.json, SynSciQA++.json in ./dataset/SynSciQA
### STEP 3: Download evaluation datasets
We evaluate our model and baselines using [ALCE](https://github.com/princeton-nlp/ALCE). To get Evaluate datasets, please run 
```bash 
bash download_test_data.sh
```
### STEP 4: Start training 
Replace the **[base model]** and the **[train/evaluate config]** below with the directory of base model and the configuration in Folder "config".
``````python
python c2cite.py --dir ./checkpoint --log_file ./logs --verbose --seed 42 --attn_impl eager --base_model [base model] --config [train/evaluate config] --device cuda:0
``````
### STEP 5: Conduct evaluation
After training process, we can conduct the evaluation step with the command below:
``````python
python c2cite.py --dir ./checkpoint --log_file ./logs --verbose --seed 42 --attn_impl eager --base_model [base model] --config [train/evaluate config] --device cuda:0 --evaluate
``````
***Note***:   **Do not** change the information in the **train config** after training step, or it won't find the right adapter.