Spaces:
Runtime error
Runtime error
Andreas Varvarigos commited on
Delete README.md
Browse files
README.md
DELETED
|
@@ -1,108 +0,0 @@
|
|
| 1 |
-
<p align="center">
|
| 2 |
-
<img src="img/litbench_interface.jpeg" alt="LitBench Interface" width="950"/>
|
| 3 |
-
</p>
|
| 4 |
-
|
| 5 |
-
<p align="center">
|
| 6 |
-
🤗 <a href="https://huggingface.co/datasets/AliMaatouk/arXiv_Topics"> arXiv Topics Dataset</a> | 🤗 <a href="https://huggingface.co/datasets/AliMaatouk/arXiv-Topics-Embeddings"> arXiv Topics Dataset Embeddings</a>
|
| 7 |
-
<br>
|
| 8 |
-
|
| 9 |
-
|
| 10 |
-
# LitBench: A Graph-Centric Large Language Model Benchmarking Framework For Literature Tasks
|
| 11 |
-
|
| 12 |
-
## Overview
|
| 13 |
-
|
| 14 |
-
LitBench is a benchmarking framework designed to retrieve, process, and fine-tune LLMs on academic literature-related tasks. It leverages the arXiv Topics dataset—which provides structured topic information for papers—and the arXiv Topic Embeddings, which contains their respective embeddings, available on Hugging Face. These embeddings are used in the retriever to find relevant papers and are also integrated into the constructed literature graph.
|
| 15 |
-
|
| 16 |
-
1. **Retrieve relevant papers**: Given a user query, LitBench finds the most relevant papers by computing **cosine similarity** between the query and available academic papers.
|
| 17 |
-
2. **Download and clean papers**: The selected papers are retrieved from **arXiv** and processed to extract structured and unstructured content.
|
| 18 |
-
3. **Construct a literature graph**: This graph contains key attributes such as:
|
| 19 |
-
- **Title, Abstract, Introduction**
|
| 20 |
-
- **Topics of the paper**
|
| 21 |
-
- **Citation sentences**
|
| 22 |
-
- **Full unstructured content (if desired)**
|
| 23 |
-
- **Edges representing citation relationships**
|
| 24 |
-
4. **Fine-tune LLMs on graph-related tasks**: The constructed graph is used to develop and benchmark LLMs on **downstream literature tasks**, including **related work generation, citation prediction, and academic text analysis**.
|
| 25 |
-
|
| 26 |
-
LitBench provides a comprehensive framework to **curate, analyze, and benchmark domain-specific LLMs** on literature-related tasks.
|
| 27 |
-
|
| 28 |
-
<p align="center">
|
| 29 |
-
<img src="img/arxiv_logo.jpeg" alt="arXiv Logo" width="220"/>
|
| 30 |
-
</p>
|
| 31 |
-
|
| 32 |
-
## Key Features:
|
| 33 |
-
|
| 34 |
-
- **Automated Literature Retrieval**: Uses **cosine similarity** to identify and download the most relevant papers from **arXiv**.
|
| 35 |
-
- **Graph Construction**: Extracts structured information from papers to build a **literature knowledge graph**.
|
| 36 |
-
- **LLM Fine-Tuning & Benchmarking**: Uses the graph to train and evaluate models on **domain-specific literature tasks**.
|
| 37 |
-
- **User Interface (UI)**: Provides an intuitive UI for user interaction, enabling seamless query input, paper retrieval, and visualization of literature graphs.
|
| 38 |
-
- **Open-Source & Scalable**: Provides an **AI agent tool** for seamless **data retrieval, graph construction, and model training**.
|
| 39 |
-
|
| 40 |
-
## Installation
|
| 41 |
-
|
| 42 |
-
### Prerequisites
|
| 43 |
-
|
| 44 |
-
Ensure you have Python 3.10 installed before proceeding with the setup.
|
| 45 |
-
|
| 46 |
-
### Setup Steps
|
| 47 |
-
|
| 48 |
-
```bash
|
| 49 |
-
# Clone the repository
|
| 50 |
-
git clone <repository_url>
|
| 51 |
-
cd LitBench
|
| 52 |
-
|
| 53 |
-
# Create a virtual environment
|
| 54 |
-
python3.10 -m venv litbench
|
| 55 |
-
|
| 56 |
-
# Activate the virtual environment
|
| 57 |
-
source litbench/bin/activate # On macOS/Linux
|
| 58 |
-
litbench\Scripts\activate # On Windows
|
| 59 |
-
|
| 60 |
-
# Install dependencies
|
| 61 |
-
pip install -r requirements.txt
|
| 62 |
-
|
| 63 |
-
# Token setup for Hugging Face
|
| 64 |
-
huggingface-cli login
|
| 65 |
-
```
|
| 66 |
-
|
| 67 |
-
## Usage
|
| 68 |
-
|
| 69 |
-
Running the Citation Graph Module
|
| 70 |
-
|
| 71 |
-
```bash
|
| 72 |
-
cd LitBench
|
| 73 |
-
source litbench/bin/activate # Activate virtual environment
|
| 74 |
-
python3.10 src/litbench_pipeline.py
|
| 75 |
-
```
|
| 76 |
-
|
| 77 |
-
## Navigating the LitBench UI
|
| 78 |
-
|
| 79 |
-
The LitBench user interface consists of two main stages: **preferences selection** and **the interactive chatbot interface**.
|
| 80 |
-
|
| 81 |
-
### **1. Setting Preferences**
|
| 82 |
-
Upon launching the interface, users are first directed to the preferences page, where they must specify:
|
| 83 |
-
- Whether to **download** new papers and construct a dataset from scratch, otherwise uses a pre-defined dataset set from the config file.
|
| 84 |
-
- Whether to **train the model** on the retrieved/predetermined dataset or use a pre-trained model from the config file.
|
| 85 |
-
|
| 86 |
-
Once preferences are set, users are directed to the chatbot interface.
|
| 87 |
-
|
| 88 |
-
### **2. Chatbot Interface**
|
| 89 |
-
After setting preferences:
|
| 90 |
-
- If **training is selected**, users will first be prompted to specify their domain of interest before proceeding.
|
| 91 |
-
- If **no training is selected**, users will be immediately prompted to provide their task prompt.
|
| 92 |
-
|
| 93 |
-
Once relevant papers are retrieved, downloaded, and cleaned (if `download=True`), and the model is fine-tuned (if training is enabled), users will be prompted to **enter their task prompt**.
|
| 94 |
-
|
| 95 |
-
### **3. Selecting a Task (Optional)**
|
| 96 |
-
The UI provides a **dropdown menu** with eight predefined literature tasks. If your task corresponds to one of these, please select it from the dropdown. Each task has a **specific input format**, which you can find in the docs/tasks/ directory.
|
| 97 |
-
|
| 98 |
-
To format your input correctly, refer to the corresponding `.md` file for each task:
|
| 99 |
-
- **Citation Sentence Generation** → [`citation_sentence.md`](docs/tasks/citation_sentence.md)
|
| 100 |
-
- **Citation Link Prediction** → [`link_pred.md`](docs/tasks/link_pred.md)
|
| 101 |
-
- **Abstract Completion** → [`abs_completion.md`](docs/tasks/abs_completion.md)
|
| 102 |
-
- **Title Generation** → [`abs_2_title.md`](docs/tasks/abs_2_title.md)
|
| 103 |
-
- **Citation Recommendation** → [`paper_retrieval.md`](docs/tasks/paper_retrieval.md)
|
| 104 |
-
- **Introduction to Abstract** → [`intro_2_abs.md`](docs/tasks/intro_2_abs.md)
|
| 105 |
-
- **Influential Papers Recommendation** → [`influential_papers.md`](docs/tasks/influential_papers.md)
|
| 106 |
-
- **Related Work Generation** → [`gen_related_work.md`](docs/tasks/gen_related_work.md)
|
| 107 |
-
|
| 108 |
-
If no task is selected, the model will run a **general inference process**, responding freely based on the user's prompt.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|