--- library_name: transformers license: apache-2.0 base_model: allenai/specter2_base tags: - generated_from_trainer metrics: - accuracy model-index: - name: results results: [] --- # 📗 SPECTER2–MAG (Multiclass Classification on MAG Level-0 Fields of Study) This model is a fine-tuned version of [allenai/specter2_base](https://huggingface.co/allenai/specter2_base) for multiclass bibliometric classification using MAG Fields of Study – Level 0 (SciDocs). It achieves the following results on the evaluation set: - Loss: 1.0598 - Accuracy: 0.8310 - Precision Micro: 0.8310 - Precision Macro: 0.8290 - Recall Micro: 0.8310 - Recall Macro: 0.8276 - F1 Micro: 0.8310 - F1 Macro: 0.8263 ## Model description This model is a fine-tuned version of SPECTER2 (`allenai/specter2_base`) adapted for multiclass classification across the 19 top-level Fields of Study (FoS) from the Microsoft Academic Graph (MAG). The model accepts the title, abstract, or title + abstract of a scientific publication and assigns it to exactly one of the MAG Level-0 domains (e.g., Biology, Chemistry, Computer Science, Engineering, Psychology). Key characteristics: * Base model: allenai/specter2_base * Task: multiclass document classification * Labels: 19 MAG Field of Study Level-0 categories * Activation: softmax * Loss: CrossEntropyLoss * Output: single best-matching FoS category MAG Level-0 represents broad disciplinary domains designed for high-level categorization of scientific documents. ## Intended uses & limitations ### Intended uses This multiclass MAG model is suitable for: - Assigning publications to **top-level scientific disciplines** - Enriching metadata in: - repositories - research output systems - funding and project datasets - bibliometric dashboards - Supporting scientometric analyses such as: - broad-discipline portfolio mapping - domain-level clustering - modeling research diversification - Classifying documents when only **title/abstract** is available The model supports inputs such as: - **title only** - **abstract only** - **title + abstract** (recommended) ### Limitations - MAG Level-0 categories are **very coarse** (e.g., *Biology*, *Medicine*, *Engineering*), and do not represent subfields. - Documents spanning multiple fields must be forced into **one** label—an inherent limitation of multiclass classification. - The training labels come from **MAG’s automatic field assignment pipeline**, not manual expert annotation. - Not suitable for: - fine-grained subdisciplines - downstream tasks requiring multilabel outputs - WoS Categories or ASJC Areas (use separate models) - clinical or regulatory decision-making Predictions should be treated as **high-level disciplinary metadata**, not detailed field classification. ## Training and evaluation data ### Source dataset: **SciDocs** Training data comes from the **SciDocs** dataset, introduced together with the original SPECTER paper: > **SciDocs** provides citation graphs, titles, abstracts, and **MAG Fields of Study** for scientific documents derived from MAG. > For this model, we use **MAG Level-0 FoS**, the 19 top-level scientific domains. Dataset characteristics: | Property | Value | |---------|-------| | Documents | ~40k scientific papers | | Labels | 19 FoS Level-0 categories | | Input fields | Abstract | | Task type | Multiclass | | Source | SciDocs (SPECTER paper) | | License | CC-BY | ## Training procedure ### Preprocessing - Input text constructed as: `abstract` - Tokenization using the SPECTER2 tokenizer - Maximum sequence length: **512 tokens** ### Model - Base model: `allenai/specter2_base` - Classification head: linear layer → softmax - Loss: **CrossEntropyLoss** ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 2e-05 - train_batch_size: 16 - eval_batch_size: 16 - seed: 42 - optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments - lr_scheduler_type: linear - num_epochs: 10 ### Training results | Training Loss | Epoch | Step | Validation Loss | Accuracy | Precision Micro | Precision Macro | Recall Micro | Recall Macro | F1 Micro | F1 Macro | |:-------------:|:-----:|:----:|:---------------:|:--------:|:---------------:|:---------------:|:------------:|:------------:|:--------:|:--------:| | 0.2603 | 1.0 | 1094 | 0.6733 | 0.8243 | 0.8243 | 0.8315 | 0.8243 | 0.8198 | 0.8243 | 0.8222 | | 0.1779 | 2.0 | 2188 | 0.6955 | 0.8240 | 0.8240 | 0.8198 | 0.8240 | 0.8203 | 0.8240 | 0.8176 | | 0.1628 | 3.0 | 3282 | 0.8130 | 0.8315 | 0.8315 | 0.8296 | 0.8315 | 0.8265 | 0.8315 | 0.8269 | | 0.1136 | 4.0 | 4376 | 0.9842 | 0.8227 | 0.8227 | 0.8254 | 0.8227 | 0.8192 | 0.8227 | 0.8205 | | 0.0666 | 5.0 | 5470 | 1.0598 | 0.8310 | 0.8310 | 0.8290 | 0.8310 | 0.8276 | 0.8310 | 0.8263 | ### Evaluation results | | precision | recall | f1-score | support | |:----------------------|------------:|---------:|-----------:|------------:| | Art | 0.654867 | 0.845714 | 0.738155 | 175 | | Biology | 0.982222 | 0.973568 | 0.977876 | 227 | | Business | 0.914894 | 0.877551 | 0.895833 | 196 | | Chemistry | 0.97449 | 0.969543 | 0.97201 | 197 | | Computer science | 0.960452 | 0.894737 | 0.926431 | 190 | | Economics | 0.816425 | 0.782407 | 0.799054 | 216 | | Engineering | 0.906103 | 0.927885 | 0.916865 | 208 | | Environmental science | 0.975369 | 0.916667 | 0.945107 | 216 | | Geography | 0.758454 | 0.912791 | 0.828496 | 172 | | Geology | 0.96729 | 0.976415 | 0.971831 | 212 | | History | 0.62987 | 0.518717 | 0.568915 | 187 | | Materials science | 0.932432 | 0.958333 | 0.945205 | 216 | | Mathematics | 0.938776 | 0.94359 | 0.941176 | 195 | | Medicine | 0.982558 | 0.923497 | 0.952113 | 183 | | Philosophy | 0.752874 | 0.748571 | 0.750716 | 175 | | Physics | 0.964824 | 0.974619 | 0.969697 | 197 | | Political science | 0.642512 | 0.661692 | 0.651961 | 201 | | Psychology | 0.806283 | 0.758621 | 0.781726 | 203 | | Sociology | 0.438889 | 0.427027 | 0.432877 | 185 | | accuracy | 0.845641 | 0.845641 | 0.845641 | 0.845641 | | macro avg | 0.842083 | 0.841681 | 0.840318 | 3751 | | weighted avg | 0.847843 | 0.845641 | 0.845311 | 3751 | ### Framework versions - Transformers 4.57.1 - Pytorch 2.8.0+cu126 - Datasets 3.6.0 - Tokenizers 0.22.1