DistilBERT Base Math Fine-Tuned (Masked Language Modeling)

This model is a fine-tuned version of distilbert-base-uncased trained on the nlile/hendrycks-MATH-benchmark dataset. It has been adapted using whole-word masking and fixed-length chunking to understand mathematical syntax, equations, numbers, and LaTeX expressions.

Model Details

  • Architecture: DistilBERT (DistilBertForMaskedLM)
  • Parameters: ~66.9 Million
  • Task: Masked Language Modeling (MLM) / Mathematical Syntax Pre-training
  • Training Environment: Kaggle (NVIDIA T4 GPU) using Hugging Face accelerate with mixed-precision (fp16)
  • Epochs Trained: 3+

Training Procedure

Hyperparameters

  • Learning Rate: $3 imes 10^{-5}$
  • Batch Size: 16
  • Max Sequence Length: 128 tokens
  • MLM Probability: 15% with a custom Whole-Word Masking (WWM) collator to prevent partial token masking of math terms and LaTeX sequences.

Evaluation Results

Evaluated on the official Hendrycks MATH test split:

  • Test Loss: [Insert your final test loss here, e.g., 1.035]
  • Test Perplexity: [Insert your final perplexity here, e.g., 2.81]

Usage Example

You can load and use this model directly with the Hugging Face pipeline for masked language modeling:

from transformers import AutoModelForMaskedLM, AutoTokenizer, pipeline

# Load model and tokenizer
tokenizer = AutoTokenizer.from_pretrained(
    "EricAI03/distilbert-base-math-finetuned-accelerate"
)
model = AutoModelForMaskedLM.from_pretrained(
    "EricAI03/distilbert-base-math-finetuned-accelerate"
)

# Create fill-mask pipeline
math_pipe = pipeline("fill-mask", model=model, tokenizer=tokenizer)

# Test prompt
prompt = "If a triangle has sides 3, 4, and [MASK], it is a right-angled triangle."
results = math_pipe(prompt, top_k=3)

for res in results:
  print(
      f"Token: '{res['token_str']}' (Score: {res['score']:.4f}) | Sentence:"
      f" {res['sequence']}"
  )
Downloads last month
63
Safetensors
Model size
67M params
Tensor type
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for EricAI03/distilbert-base-math-finetuned-accelerate

Finetuned
(12,902)
this model

Dataset used to train EricAI03/distilbert-base-math-finetuned-accelerate