Andreas Varvarigos commited on
Commit
ec5f3b0
·
verified ·
1 Parent(s): 9860d5e

Delete README.md

Browse files
Files changed (1) hide show
  1. README.md +0 -108
README.md DELETED
@@ -1,108 +0,0 @@
1
- <p align="center">
2
- <img src="img/litbench_interface.jpeg" alt="LitBench Interface" width="950"/>
3
- </p>
4
-
5
- <p align="center">
6
- 🤗 <a href="https://huggingface.co/datasets/AliMaatouk/arXiv_Topics"> arXiv Topics Dataset</a>&nbsp| 🤗 <a href="https://huggingface.co/datasets/AliMaatouk/arXiv-Topics-Embeddings"> arXiv Topics Dataset Embeddings</a>
7
- <br>
8
-
9
-
10
- # LitBench: A Graph-Centric Large Language Model Benchmarking Framework For Literature Tasks
11
-
12
- ## Overview
13
-
14
- LitBench is a benchmarking framework designed to retrieve, process, and fine-tune LLMs on academic literature-related tasks. It leverages the arXiv Topics dataset—which provides structured topic information for papers—and the arXiv Topic Embeddings, which contains their respective embeddings, available on Hugging Face. These embeddings are used in the retriever to find relevant papers and are also integrated into the constructed literature graph.
15
-
16
- 1. **Retrieve relevant papers**: Given a user query, LitBench finds the most relevant papers by computing **cosine similarity** between the query and available academic papers.
17
- 2. **Download and clean papers**: The selected papers are retrieved from **arXiv** and processed to extract structured and unstructured content.
18
- 3. **Construct a literature graph**: This graph contains key attributes such as:
19
- - **Title, Abstract, Introduction**
20
- - **Topics of the paper**
21
- - **Citation sentences**
22
- - **Full unstructured content (if desired)**
23
- - **Edges representing citation relationships**
24
- 4. **Fine-tune LLMs on graph-related tasks**: The constructed graph is used to develop and benchmark LLMs on **downstream literature tasks**, including **related work generation, citation prediction, and academic text analysis**.
25
-
26
- LitBench provides a comprehensive framework to **curate, analyze, and benchmark domain-specific LLMs** on literature-related tasks.
27
-
28
- <p align="center">
29
- <img src="img/arxiv_logo.jpeg" alt="arXiv Logo" width="220"/>
30
- </p>
31
-
32
- ## Key Features:
33
-
34
- - **Automated Literature Retrieval**: Uses **cosine similarity** to identify and download the most relevant papers from **arXiv**.
35
- - **Graph Construction**: Extracts structured information from papers to build a **literature knowledge graph**.
36
- - **LLM Fine-Tuning & Benchmarking**: Uses the graph to train and evaluate models on **domain-specific literature tasks**.
37
- - **User Interface (UI)**: Provides an intuitive UI for user interaction, enabling seamless query input, paper retrieval, and visualization of literature graphs.
38
- - **Open-Source & Scalable**: Provides an **AI agent tool** for seamless **data retrieval, graph construction, and model training**.
39
-
40
- ## Installation
41
-
42
- ### Prerequisites
43
-
44
- Ensure you have Python 3.10 installed before proceeding with the setup.
45
-
46
- ### Setup Steps
47
-
48
- ```bash
49
- # Clone the repository
50
- git clone <repository_url>
51
- cd LitBench
52
-
53
- # Create a virtual environment
54
- python3.10 -m venv litbench
55
-
56
- # Activate the virtual environment
57
- source litbench/bin/activate # On macOS/Linux
58
- litbench\Scripts\activate # On Windows
59
-
60
- # Install dependencies
61
- pip install -r requirements.txt
62
-
63
- # Token setup for Hugging Face
64
- huggingface-cli login
65
- ```
66
-
67
- ## Usage
68
-
69
- Running the Citation Graph Module
70
-
71
- ```bash
72
- cd LitBench
73
- source litbench/bin/activate # Activate virtual environment
74
- python3.10 src/litbench_pipeline.py
75
- ```
76
-
77
- ## Navigating the LitBench UI
78
-
79
- The LitBench user interface consists of two main stages: **preferences selection** and **the interactive chatbot interface**.
80
-
81
- ### **1. Setting Preferences**
82
- Upon launching the interface, users are first directed to the preferences page, where they must specify:
83
- - Whether to **download** new papers and construct a dataset from scratch, otherwise uses a pre-defined dataset set from the config file.
84
- - Whether to **train the model** on the retrieved/predetermined dataset or use a pre-trained model from the config file.
85
-
86
- Once preferences are set, users are directed to the chatbot interface.
87
-
88
- ### **2. Chatbot Interface**
89
- After setting preferences:
90
- - If **training is selected**, users will first be prompted to specify their domain of interest before proceeding.
91
- - If **no training is selected**, users will be immediately prompted to provide their task prompt.
92
-
93
- Once relevant papers are retrieved, downloaded, and cleaned (if `download=True`), and the model is fine-tuned (if training is enabled), users will be prompted to **enter their task prompt**.
94
-
95
- ### **3. Selecting a Task (Optional)**
96
- The UI provides a **dropdown menu** with eight predefined literature tasks. If your task corresponds to one of these, please select it from the dropdown. Each task has a **specific input format**, which you can find in the docs/tasks/ directory.
97
-
98
- To format your input correctly, refer to the corresponding `.md` file for each task:
99
- - **Citation Sentence Generation** → [`citation_sentence.md`](docs/tasks/citation_sentence.md)
100
- - **Citation Link Prediction** → [`link_pred.md`](docs/tasks/link_pred.md)
101
- - **Abstract Completion** → [`abs_completion.md`](docs/tasks/abs_completion.md)
102
- - **Title Generation** → [`abs_2_title.md`](docs/tasks/abs_2_title.md)
103
- - **Citation Recommendation** → [`paper_retrieval.md`](docs/tasks/paper_retrieval.md)
104
- - **Introduction to Abstract** → [`intro_2_abs.md`](docs/tasks/intro_2_abs.md)
105
- - **Influential Papers Recommendation** → [`influential_papers.md`](docs/tasks/influential_papers.md)
106
- - **Related Work Generation** → [`gen_related_work.md`](docs/tasks/gen_related_work.md)
107
-
108
- If no task is selected, the model will run a **general inference process**, responding freely based on the user's prompt.