| # ERC Classifiers | |
| This repository contains a model trained for multi-label classification of scientific papers in the ERC (European Research Council) context. The model predicts multiple categories for a paper, such as its research domain or topic, based on the abstract and title. | |
| ## Model Description | |
| The model is based on **SPECTER** (a transformer-based model pre-trained on scientific literature), fine-tuned for **multi-label classification** on a dataset of scientific papers. The model classifies papers into several categories, which are defined by the **ERC categories**. The fine-tuned model is trained to predict these categories given the title and abstract of each paper. | |
| ### Preprocessing | |
| The preprocessing pipeline involves: | |
| 1. **Data Loading**: Papers are loaded from a Parquet file containing the title, abstract, and their respective categories. | |
| 2. **Label Cleaning**: Labels (categories) are processed to remove any unnecessary information (like content within parentheses). | |
| 3. **Label Encoding**: Categories are transformed into a binary matrix using the **MultiLabelBinarizer** from scikit-learn. Each category corresponds to a column, and the value is `1` if the paper belongs to that category, `0` otherwise. | |
| 4. **Statistics and Visualization**: Basic statistics and visualizations, such as label distributions, are generated to help understand the dataset better. | |
| ### Training | |
| The model is fine-tuned on the preprocessed dataset using the following setup: | |
| * **Base Model**: The model uses the `allenai/specter` transformer as the base model for sequence classification. | |
| * **Optimizer**: AdamW optimizer with a learning rate of `5e-5` is used. | |
| * **Loss Function**: Binary Cross-Entropy with logits (`BCEWithLogitsLoss`) is employed, as the task is multi-label classification. | |
| * **Epochs**: The model is trained for **5 epochs** with a batch size of 4. | |
| * **Training Data**: The model is trained on a processed dataset stored in `train_ready.parquet`. | |
| ### Evaluation | |
| The model is evaluated using both **single-label** and **multi-label** metrics: | |
| #### Single-Label Evaluation | |
| * **Accuracy**: The accuracy is measured by checking how often the true label appears in the predicted labels. | |
| * **Precision, Recall, F1**: These metrics are calculated for each class and averaged for the entire dataset. | |
| #### Multi-Label Evaluation | |
| * **Micro and Macro Metrics**: Precision, recall, and F1 scores are computed using both micro-averaging (overall performance) and macro-averaging (performance per label). | |
| * **Label Frequency Plot**: A plot showing the frequency distribution of labels in the test set. | |
| * **Top and Bottom F1 Plot**: A plot visualizing the top and bottom labels based on their F1 scores. | |
| ## Dataset | |
| The dataset consists of scientific papers, each with the following columns: | |
| * **title**: The title of the paper. | |
| * **abstract**: The abstract of the paper. | |
| * **label**: A list of categories (labels) assigned to the paper. | |
| The dataset is preprocessed and stored in a `train_ready.parquet` file. | |
| ## Files | |
| * `config.json`: Model configuration file. | |
| * `model.safetensors`: Saved fine-tuned model weights. | |
| * `tokenizer.json`: Tokenizer configuration for the fine-tuned model. | |
| * `tokenizer_config.json`: Tokenizer settings. | |
| * `special_tokens_map.json`: Special tokens used by the tokenizer. | |
| * `vocab.txt`: Vocabulary file for the fine-tuned tokenizer. | |
| ## Usage | |
| To use the model, follow these steps: | |
| 1. **Install Dependencies**: | |
| ```bash | |
| pip install transformers torch datasets | |
| ``` | |
| 2. **Load the Model and Tokenizer**: | |
| ```python | |
| from transformers import AutoModelForSequenceClassification, AutoTokenizer | |
| model_name = "SIRIS-Lab/erc-classifiers" | |
| # Load fine-tuned model and tokenizer | |
| model = AutoModelForSequenceClassification.from_pretrained(model_name) | |
| tokenizer = AutoTokenizer.from_pretrained(model_name) | |
| ``` | |
| 3. **Use the Model for Prediction**: | |
| ```python | |
| # Example paper title and abstract | |
| text = "Example title and abstract of a scientific paper." | |
| # Tokenize the input text | |
| inputs = tokenizer(text, return_tensors="pt", padding=True, truncation=True) | |
| # Make predictions | |
| with torch.no_grad(): | |
| logits = model(**inputs).logits | |
| # Apply sigmoid activation to get probabilities | |
| probabilities = torch.sigmoid(logits) | |
| # Get predicted labels (threshold at 0.5) | |
| predicted_labels = (probabilities >= 0.5).long().cpu().numpy() | |
| print(predicted_labels) | |
| ``` | |
| ## Conclusion | |
| This model provides an efficient solution for classifying scientific papers into multiple categories based on their content. It uses state-of-the-art transformer-based techniques and is fine-tuned on a real-world dataset of ERC-related scientific papers. | |