bart-base-dwikipedia-simplification

facebook/bart-base fine-tuned on D-Wikipedia (Sun et al., EMNLP 2021) for document-level English text simplification. Unlike a sentence-level simplifier, it rewrites a whole multi-sentence passage at once and is expected to delete, merge, split and reorder sentences — so its output has no position-by-position correspondence to its input.

Trained as part of a bachelor thesis on automated simplification of everyday English web text, and served as the document model by the project's backend: https://github.com/yyvs/simple-website

⚠️ Preliminary checkpoint. This is a reduced-scope run (20,000-document train slice, 2 epochs), and it has never been evaluated on a document-level benchmark. See Status before using or citing it.

⚠️ Input and output are lowercased and PTB-pre-tokenized

This is the most important usage detail, and getting it wrong degrades output badly.

D-Wikipedia is fully lowercased, PTB-pre-tokenized (writer , intellectual, women 's, `` the second sex '') and one document per line, with zero newline characters inside any document body (verified across 3,000 documents). The model both consumes and emits that convention.

Measured consequences of ignoring it:

• Feeding ordinary mixed-case prose produced a factual hallucination ("northern Netherlands" → "northern hemisphere") that the lowercased, corpus-style input did not.
• Supplying \n as a structural separator — the intuitive way to convey document structure — produced the worst output of every variant tested, losing more content and hallucinating more than either alternative, because the model has never seen a newline in training.

Raw output looks like achtkarspelen is a municipality in friesland . — you must de-normalize it before display. The companion project implements the matched normalize/de-normalize pair in backend/document_text.py: it re-attaches punctuation, restores quotes, then recovers casing in two passes (proper nouns and acronyms recovered from the source text, since the model only ever emits lowercase, followed by sentence-initial capitals).

Usage

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tok = AutoTokenizer.from_pretrained("yunvs/bart-base-dwikipedia-simplification")
model = AutoModelForSeq2SeqLM.from_pretrained("yunvs/bart-base-dwikipedia-simplification")

# Normalize to corpus style first: lowercase, PTB-tokenize, no newlines.
doc = "achtkarspelen is a municipality in the northern netherlands . it had a population of 27,944 in 2017 ."

inputs = tok(doc, return_tensors="pt", truncation=True, max_length=512)
out = model.generate(**inputs, max_length=512, num_beams=4,
                     no_repeat_ngram_size=3, repetition_penalty=1.2)
print(tok.decode(out[0], skip_special_tokens=True))
# -> lowercase, pre-tokenized output; de-normalize before displaying

Input past 512 tokens is truncated, not chunked — the overflow is silently dropped rather than simplified. Chunk below the limit yourself. A real Wikipedia article measured 2,459 BART tokens, i.e. ~79% would have been discarded if fed whole.

Do not pack input to fill 512. The training median is 120 tokens and 68% of training documents are ≤200, while only 8% are ≥450. A lone 100-token section is more representative of the training distribution than a merged 500-token one — 512 is a ceiling for splitting, never a target for packing.

Do not include headings. When a heading was included in a section's text, the model echoed it back into the body output. Headings are also far shorter than anything it saw in training.

Training

Base model	facebook/bart-base
Dataset	D-Wikipedia (Sun et al., EMNLP 2021), 20,000-document train slice
Scope	"reduced" — not a full-corpus run
Epochs	2
Batch size	2, with gradient_accumulation_steps=2
Gradient checkpointing	Enabled
Precision	bf16
Device	Apple Silicon (MPS)
Learning rate	3e-5
Weight decay	0.01
Max length	512 tokens (source p95 ≈ 499)
Seed	42

The small batch size and gradient checkpointing are MPS accommodations — on Apple Silicon, GPU and host memory are one unified pool — not modeling choices.

max_length=512 was measured, not inherited from BART's 1024-token ceiling: source documents mean ≈172 tokens, p95 ≈499, p99 ≈742; targets mean ≈98. 512 covers ~p95 while cutting self-attention's O(n²) memory cost to a quarter of the full ceiling.

Observed training loss (step | train | validation): 1250 | 0.7859 | 0.1956 — 2500 | 0.3665 | 0.1907.

Status: read this before citing it

No document-level evaluation result exists for this model.

The correct benchmark — D-Wikipedia's own test split scored with D-SARI, LENS and BERTScore — is implemented in the project but has not been run against this checkpoint.

What was run is a mismatched evaluation, reported here only for honesty about what exists. This checkpoint was scored on ASSET (359 isolated sentences), which is the wrong benchmark for a document-trained model:

	SARI ↑	BLEU	FKGL ↓
This model (on ASSET — wrong benchmark)	33.54	50.83	7.71
facebook/bart-base zero-shot	21.34	89.89	10.02

The direction is encouraging (SARI up, FKGL down, BLEU down — the same pattern as the sentence-level model) and suggests some transfer from document-level training to sentence-level output. It is not a measurement of document-level simplification quality and must not be quoted as one.

Limitations

• Preliminary and reduced-scope. A full-corpus run has not completed.
• Unevaluated on its actual task, as above.
• Output is not aligned to input. By design it deletes, merges and reorders — you cannot map a sentence in the output back to a sentence in the input. Any UI must replace a whole block, not individual paragraphs.
• Requires corpus-style input and output post-processing (see above). This is not cosmetic; skipping it caused a measured factual hallucination.
• Silent truncation past 512 tokens.
• No human evaluation, and only an unexamined qualitative spot-check on a handful of real test-split documents.
• English only, encyclopedic register.

Training data provenance and licence

D-Wikipedia is derived from English Wikipedia and Simple English Wikipedia (CC BY-SA). This model is released under CC BY-SA 4.0 accordingly.

Sun, R., Jin, H., & Wan, X. (2021). Document-Level Text Simplification: Dataset, Criteria and Baseline. EMNLP 2021.

Citation

Produced for a bachelor thesis (2026). Please cite the project repository: https://github.com/yyvs/simple-website