| | --- |
| | {} |
| | --- |
| | # ERC Classifiers |
| |
|
| | This repository contains a model trained for multi-label classification of scientific papers in the ERC (European Research Council) context. The model predicts multiple categories for a paper, such as its research domain or topic, based on the abstract and title. |
| |
|
| | ## Model Description |
| |
|
| | The model is based on **SPECTER** (a transformer-based model pre-trained on scientific literature), fine-tuned for **multi-label classification** on a dataset of scientific papers. The model classifies papers into several categories, which are defined by the **ERC categories**. The fine-tuned model is trained to predict these categories given the title and abstract of each paper. |
| |
|
| | ### Preprocessing |
| |
|
| | The preprocessing pipeline involves: |
| |
|
| | 1. **Data Loading**: Papers are loaded from a Parquet file containing the title, abstract, and their respective categories. |
| | 2. **Label Cleaning**: Labels (categories) are processed to remove any unnecessary information (like content within parentheses). |
| | 3. **Label Encoding**: Categories are transformed into a binary matrix using the **MultiLabelBinarizer** from scikit-learn. Each category corresponds to a column, and the value is `1` if the paper belongs to that category, `0` otherwise. |
| | 4. **Statistics and Visualization**: Basic statistics and visualizations, such as label distributions, are generated to help understand the dataset better. |
| |
|
| | ### Training |
| |
|
| | The model is fine-tuned on the preprocessed dataset using the following setup: |
| |
|
| | * **Base Model**: The model uses the `allenai/specter` transformer as the base model for sequence classification. |
| | * **Optimizer**: AdamW optimizer with a learning rate of `5e-5` is used. |
| | * **Loss Function**: Binary Cross-Entropy with logits (`BCEWithLogitsLoss`) is employed, as the task is multi-label classification. |
| | * **Epochs**: The model is trained for **5 epochs** with a batch size of 4. |
| | * **Training Data**: The model is trained on a processed dataset stored in `train_ready.parquet`. |
| |
|
| | ### Evaluation |
| |
|
| | The model is evaluated using both **single-label** and **multi-label** metrics: |
| |
|
| | #### Single-Label Evaluation |
| |
|
| | * **Accuracy**: The accuracy is measured by checking how often the true label appears in the predicted labels. |
| | * **Precision, Recall, F1**: These metrics are calculated for each class and averaged for the entire dataset. |
| |
|
| | #### Multi-Label Evaluation |
| |
|
| | * **Micro and Macro Metrics**: Precision, recall, and F1 scores are computed using both micro-averaging (overall performance) and macro-averaging (performance per label). |
| | * **Label Frequency Plot**: A plot showing the frequency distribution of labels in the test set. |
| | * **Top and Bottom F1 Plot**: A plot visualizing the top and bottom labels based on their F1 scores. |
| |
|
| | ## Dataset |
| |
|
| | The dataset consists of scientific papers, each with the following columns: |
| |
|
| | * **title**: The title of the paper. |
| | * **abstract**: The abstract of the paper. |
| | * **label**: A list of categories (labels) assigned to the paper. |
| |
|
| | The dataset is preprocessed and stored in a `train_ready.parquet` file. |
| |
|
| | ## Files |
| |
|
| | * `config.json`: Model configuration file. |
| | * `model.safetensors`: Saved fine-tuned model weights. |
| | * `tokenizer.json`: Tokenizer configuration for the fine-tuned model. |
| | * `tokenizer_config.json`: Tokenizer settings. |
| | * `special_tokens_map.json`: Special tokens used by the tokenizer. |
| | * `vocab.txt`: Vocabulary file for the fine-tuned tokenizer. |
| |
|
| | ## Usage |
| |
|
| | To use the model, follow these steps: |
| |
|
| | 1. **Install Dependencies**: |
| |
|
| | ```bash |
| | pip install transformers torch datasets |
| | ``` |
| |
|
| | 2. **Load the Model and Tokenizer**: |
| |
|
| | ```python |
| | from transformers import AutoModelForSequenceClassification, AutoTokenizer |
| | |
| | model_name = "SIRIS-Lab/erc-classifiers" |
| | |
| | # Load fine-tuned model and tokenizer |
| | model = AutoModelForSequenceClassification.from_pretrained(model_name) |
| | tokenizer = AutoTokenizer.from_pretrained(model_name) |
| | ``` |
| |
|
| | 3. **Use the Model for Prediction**: |
| |
|
| | ```python |
| | # Example paper title and abstract |
| | text = "Example title and abstract of a scientific paper." |
| | |
| | # Tokenize the input text |
| | inputs = tokenizer(text, return_tensors="pt", padding=True, truncation=True) |
| | |
| | # Make predictions |
| | with torch.no_grad(): |
| | logits = model(**inputs).logits |
| | |
| | # Apply sigmoid activation to get probabilities |
| | probabilities = torch.sigmoid(logits) |
| | |
| | # Get predicted labels (threshold at 0.5) |
| | predicted_labels = (probabilities >= 0.5).long().cpu().numpy() |
| | print(predicted_labels) |
| | ``` |
| |
|
| | ## Conclusion |
| |
|
| | This model provides an efficient solution for classifying scientific papers into multiple categories based on their content. It uses state-of-the-art transformer-based techniques and is fine-tuned on a real-world dataset of ERC-related scientific papers. |
| |
|
