---
language:
- en
tags:
- masked-language-modeling
- bert
- vminibert
- rope
- rmsnorm
- geglu
- pytorch
- fill-mask
- from-scratch
license: mit
pipeline_tag: fill-mask
datasets:
- wikitext
- bookcorpusopen
- HuggingFaceFW/fineweb
- allenai/c4
---

# vminibert-108M

`vminibert-108M` is a 108.6M-parameter Transformer encoder trained **from scratch** as a Masked Language Model on a blended 4-domain corpus (WikiText-103, BookCorpusOpen, FineWeb, and C4) using dynamic Whole-Word Masking (WWM).

Rather than a stock BERT reimplementation, it uses a modern LLaMA-style encoder recipe:

- **RoPE** (Rotary Position Embeddings) instead of learned absolute positions
- **RMSNorm** instead of LayerNorm
- **GeGLU** feed-forward blocks instead of a plain GELU MLP

## Model Details

| | |
| :--- | :--- |
| **Parameters** | 108,615,984 |
| **Hidden size** | 768 |
| **Attention heads** | 12 |
| **Encoder layers** | 12 |
| **Intermediate size (GeGLU)** | 2048 |
| **Vocabulary size** | 30,000 (WordPiece) |
| **Max sequence length** | 32 |
| **Positional encoding** | RoPE |
| **Normalization** | RMSNorm |
| **Activation** | GeGLU |
| **Weight tying** | Output projection tied to input embeddings |

## Training Data

Trained on ~250k samples blended from four domains:

| Source | Domain |
| :--- | :--- |
| WikiText-103 | Encyclopedic |
| BookCorpusOpen | Long-form narrative |
| FineWeb | General web text |
| C4 | Cleaned web crawl |

Masking used dynamic Whole-Word Masking at a 15% rate.

## Evaluation

Evaluated on official unseen test splits with standard 15% WWM:

| Benchmark | Split | Masked Tokens | Loss | Perplexity | Top-1 Acc | Top-3 Acc | Top-5 Acc | MRR@10 |
| :--- | :--- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| Salesforce/wikitext-2-raw-v1 | test | 8,552 | 4.0323 | 56.39 | 43.15% | 53.86% | 57.96% | — |
| Salesforce/wikitext-103-raw-v1 | test | 8,515 | 4.0378 | 56.70 | 42.82% | 53.55% | 58.06% | 49.29% |

*MRR@10 for wikitext-2 is not yet computed — fill in or remove the column before publishing.*

## Usage

```python
from transformers import AutoTokenizer, AutoModelForMaskedLM, pipeline

model_id = "vprojectx/VMiniBert"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForMaskedLM.from_pretrained(model_id, trust_remote_code=True)

fill_mask = pipeline("fill-mask", model=model, tokenizer=tokenizer)
results = fill_mask("The capital of France is [MASK].")

for res in results:
    print(f"{res['token_str']:<12} {res['score']:.4f}")
```

## Limitations & Intended Use

- **Context window:** pretrained and evaluated at 32-token sequences only; behavior beyond that is untested. This is the single biggest caveat for anyone evaluating this model — flag it up front, not just in a bullet list.
- **Corpus size:** ~250k samples is modest for a from-scratch LM; expect it to need task-specific fine-tuning for downstream use rather than strong zero-shot performance.
- **Intended use:** a lightweight, fast MLM encoder for fine-tuning (classification, NER, masked-token extraction) and as a reference implementation of a modern (RoPE/RMSNorm/GeGLU) BERT-style architecture trained end-to-end from scratch.
- **Not intended for:** long-document tasks, production deployment without fine-tuning, or use cases requiring calibrated confidence at scale (no calibration study was done).

## Citation

If you use this model, please cite it as:

```bibtex
@misc{vminibert2026,
  author = {vprojectx},
  title = {vminibert-108M: A From-Scratch BERT-Style Encoder with RoPE, RMSNorm, and GeGLU},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/vprojectx/VMiniBert}
}
```