gec-bart-base-s1

facebook/bart-base fine-tuned for English grammatical error correction (GEC) by the CS4248 GEC project. This is run bart-base.s1 in the project repository (experiments/training/bart-base.s1/).

Training

Stage 1: BEA-2019 train (FCE train+dev, Lang-8, NUCLE, W&I A/B/C), errorful pairs only, 561,338 pairs. There is no stage 2: this is the stage-1-only model.

Run Stage Learning rate Batch Epochs Best step
bart-base.s1 Stage 1 5e-05 128 3 13,155

Settings common to all stages:

  • AdamW with linear decay
  • weight decay 0.01, label smoothing 0.1, dropout 0.1, seed 42
  • bf16, max length 128 tokens

The checkpoint is the one with the best ERRANT F0.5 on dev_tune. Training ran on an NVIDIA H100 NVL with torch 2.11.0+cu128 and transformers 4.46.3. Git commit 8371a55.

Results

ERRANT P / R / F0.5 (%), beam 5:

Split Score
dev_test: 2,384 W&I+LOCNESS dev sentences, never used for choosing checkpoints 49.0 / 30.8 / 43.8
full W&I+LOCNESS dev (BEA-19 dev, 4,384 sentences) 49.6 / 31.1 / 44.4

The dev_tune/dev_test split, the predictions and the scores are in the project repository.

Usage

The project repository's scripts take this repository name as --model. For example:

python -m training.evaluate --model CS4248-GEC/gec-bart-base-s1 --split dev_test --out out/

Using transformers directly:

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

repo = "CS4248-GEC/gec-bart-base-s1"          # private: run `hf auth login` first
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSeq2SeqLM.from_pretrained(repo)

source = "She go to school yesterday ."   # tokenized like the corpora (It 's, do n't)
inputs = tokenizer([source], return_tensors="pt")
out = model.generate(**inputs, num_beams=5, max_new_tokens=160, no_repeat_ngram_size=0)
print(tokenizer.decode(out[0], skip_special_tokens=True))

Notes:

  • Inputs are tokenized: the model reads and writes text tokenized like the BEA-2019 corpora ("It 's", "do n't", spaces before punctuation).
  • No prefix: BART takes the source sentence as it is.
  • Always pass no_repeat_ngram_size=0: bart-base inherits 3 from its summarization defaults, which stops it copying repeated phrases. The saved generation_config.json already sets 0.

Verification

model.safetensors sha256: 40b41277a8bf99c76009d5b6b7939fbdb3d118593bc87d64381e72da507e8986

To reproduce the scores, run python -m training.evaluate --model CS4248-GEC/gec-bart-base-s1 --split dev_test --out check/ and compare the result with experiments/training/bart-base.s1/dev_test.scores.json. Other hardware may decode a few sentences differently.

Data and licence

Trained on the BEA-2019 shared-task data (FCE, Lang-8, NUCLE, W&I+LOCNESS), whose licences restrict redistribution and commercial use. This model is private to the CS4248-GEC organization, for course use only.

Downloads last month
7
Safetensors
Model size
0.1B params
Tensor type
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for CS4248-GEC/gec-bart-base-s1

Finetuned
(527)
this model