YAML Metadata Warning:empty or missing yaml metadata in repo card
Check out the documentation for more information.
XpertGPT Fine-Tuning Pipeline
This repository contains the fine-tuning pipeline and downstream evaluation results for the XpertGPT model (anonym5035/swi_glu_sw_64_16_8_4_xpert_gpt) trained for the BabyLM 2026 Strict-Small track.
The pipeline (finetune_eval.py) is designed to fine-tune the causal pre-trained model on 9 diverse sequence classification and multiple-choice tasks, bypassing autoregressive generation in favor of classification heads.
Core Hyperparameters (Global)
Across all tasks, the following core training configurations are enforced:
| Hyperparameter | Value | Notes |
|---|---|---|
| Optimizer | AdamW |
betas=(0.9, 0.999), eps=1e-8 |
| Weight Decay | 0.01 |
Lighter than pretraining (0.1) for small dataset finetuning |
| LR Schedule | Linear Decay | Preceded by a linear warmup phase |
| Warmup Ratio | 0.06 |
Warmup for 6% of total training steps |
| Gradient Clipping | 1.0 |
Matches pretraining |
| Max Sequence Len | 128 |
Truncated from 512 for efficiency on short tasks |
| Pooling Method | Exact Last Token | (mask * arange).argmax(dim=1) to ensure robust extraction |
| Hardware | 2x T4 GPUs | nn.DataParallel scales batch sizes automatically |
Results & Task Hyperparameters
Because the datasets range from massive (105K examples) to tiny (250 examples), hyperparameters are bucketed by dataset size to prevent overfitting.
| Task | Hugging Face Dataset | Type | Dataset Size | Random Guess | GPT-2 Baseline | XpertGPT | Epochs | Learning Rate | Base Batch Size |
|---|---|---|---|---|---|---|---|---|---|
| QNLI | nyu-mll/glue |
2-way Classification | Large (~105K) | 50.0% | 78.1% | 66.4% | 3 | 2e-5 |
16 |
| SWAG | allenai/swag |
4-way Multiple Choice | Large (~73K) | 25.0% | 32.9% | 35.9% | 3 | 2e-5 |
16 |
| SST-2 | stanfordnlp/sst2 |
2-way Classification | Large (~67K) | 50.0% | 85.8% | 83.5% | 3 | 2e-5 |
16 |
| WinoGrande | allenai/winogrande |
2-way Multiple Choice | Large (~40K) | 50.0% | 51.7% | 51.5% | 3 | 2e-5 |
16 |
| HellaSwag | Rowan/hellaswag |
4-way Multiple Choice | Large (~40K) | 25.0% | 30.0% | 30.3% | 3 | 2e-5 |
16 |
| Social IQa | allenai/social_i_qa |
3-way Multiple Choice | Large (~33K) | 33.3% | N/A | N/A | 3 | 2e-5 |
16 |
| CommonsenseQA | tau/commonsense_qa |
5-way Multiple Choice | Medium (~9.7K) | 20.0% | 22.6% | 24.0% | 8 | 3e-5 |
16 |
| CoLA | nyu-mll/glue |
2-way Classification | Medium (~8.5K) | 50.0% | 69.1% | 68.9% | 5 | 3e-5 |
16 |
| OpenBookQA | allenai/openbookqa |
4-way Multiple Choice | Medium (~5K) | 25.0% | 49.2% | 42.2% | 5 | 3e-5 |
16 |
| ARC-Easy | allenai/ai2_arc |
Var Multiple Choice | Tiny (~2.3K) | ~25.0% | 33.0% | 32.2% | 8 | 3e-5 |
16 |
| COPA | pkavumba/balanced-copa |
2-way Multiple Choice | Tiny (~400) | 50.0% | 46.8% | 56.0% | 5 | 5e-5 |
8 |
| CommitmentBank | aps/super_glue |
3-way Classification | Tiny (~250) | 33.3% | 76.8% | 60.7% | 5 | 5e-5 |
8 |
(Note: The "Base Batch Size" is per-GPU. Under a 2x T4 DataParallel setup, the effective batch sizes become 32 and 16 respectively).
Outputs
The script uploads two artifacts back to this repository upon completion:
results/finetune_results.json: The highest validation accuracy achieved for each dataset.results/training_history.json: Epoch-by-epoch tracking oftrain_loss,val_loss, andval_accfor advanced overfitting analysis.
Kaggle Execution
To execute this pipeline inside a Kaggle notebook (using the T4 x2 accelerator):
# 1. Install dependencies
!pip install transformers datasets huggingface_hub torch tqdm
# 2. Download the fine-tuning script
!wget -O finetune_eval.py https://huggingface.co/anonym5035/finetuning_original_model/raw/main/finetune_eval.py
# 3. Run the pipeline and push results to Hugging Face
!python finetune_eval.py --hf_token YOUR_HF_TOKEN_HERE