README.md

---
license: cc-by-sa-4.0
language:
  - en
base_model: facebook/bart-base
# See the sentence model's card for why pipeline_tag is left unset: the Hub rejects
# "text2text-generation" and mislabels the model as decoder-only without it.
tags:
  - text-simplification
  - document-simplification
  - accessibility
  - bart
datasets:
  - d-wikipedia
---

# bart-base-dwikipedia-simplification

`facebook/bart-base` fine-tuned on **D-Wikipedia** (Sun et al., EMNLP 2021) for
**document-level** English text simplification. Unlike a sentence-level simplifier, it
rewrites a whole multi-sentence passage at once and is *expected* to delete, merge,
split and reorder sentences — so its output has **no position-by-position
correspondence to its input**.

Trained as part of a bachelor thesis on automated simplification of everyday English
web text, and served as the `document` model by the project's backend:
<https://github.com/yyvs/simple-website>

> ⚠️ **Preliminary checkpoint.** This is a *reduced-scope* run (20,000-document train
> slice, 2 epochs), and it has **never been evaluated on a document-level benchmark**.
> See [Status](#status-read-this-before-citing-it) before using or citing it.

## ⚠️ Input and output are lowercased and PTB-pre-tokenized

This is the most important usage detail, and getting it wrong degrades output badly.

D-Wikipedia is fully lowercased, PTB-pre-tokenized (`writer , intellectual`,
`women 's`, ``` `` the second sex '' ```) and one document per line, with **zero
newline characters inside any document body** (verified across 3,000 documents). The
model both **consumes and emits** that convention.

Measured consequences of ignoring it:

- Feeding ordinary mixed-case prose produced a **factual hallucination**
  ("northern Netherlands" → "northern hemisphere") that the lowercased, corpus-style
  input did not.
- Supplying `\n` as a structural separator — the intuitive way to convey document
  structure — produced the **worst output of every variant tested**, losing more
  content and hallucinating more than either alternative, because the model has never
  seen a newline in training.

Raw output looks like `achtkarspelen is a municipality in friesland .` — you must
de-normalize it before display. The companion project implements the matched
normalize/de-normalize pair in
[`backend/document_text.py`](https://github.com/yyvs/simple-website/blob/main/backend/document_text.py):
it re-attaches punctuation, restores quotes, then recovers casing in two passes
(proper nouns and acronyms recovered from the *source* text, since the model only ever
emits lowercase, followed by sentence-initial capitals).

## Usage

```python
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tok = AutoTokenizer.from_pretrained("yunvs/bart-base-dwikipedia-simplification")
model = AutoModelForSeq2SeqLM.from_pretrained("yunvs/bart-base-dwikipedia-simplification")

# Normalize to corpus style first: lowercase, PTB-tokenize, no newlines.
doc = "achtkarspelen is a municipality in the northern netherlands . it had a population of 27,944 in 2017 ."

inputs = tok(doc, return_tensors="pt", truncation=True, max_length=512)
out = model.generate(**inputs, max_length=512, num_beams=4,
                     no_repeat_ngram_size=3, repetition_penalty=1.2)
print(tok.decode(out[0], skip_special_tokens=True))
# -> lowercase, pre-tokenized output; de-normalize before displaying
```

**Input past 512 tokens is truncated, not chunked** — the overflow is silently dropped
rather than simplified. Chunk below the limit yourself. A real Wikipedia article
measured 2,459 BART tokens, i.e. ~79% would have been discarded if fed whole.

**Do not pack input to fill 512.** The training median is 120 tokens and 68% of
training documents are ≤200, while only 8% are ≥450. A lone 100-token section is
*more* representative of the training distribution than a merged 500-token one — 512
is a ceiling for splitting, never a target for packing.

**Do not include headings.** When a heading was included in a section's text, the model
**echoed it back into the body output**. Headings are also far shorter than anything it
saw in training.

## Training

| | |
|---|---|
| Base model | `facebook/bart-base` |
| Dataset | D-Wikipedia (Sun et al., EMNLP 2021), 20,000-document train slice |
| Scope | **"reduced"** — not a full-corpus run |
| Epochs | 2 |
| Batch size | 2, with `gradient_accumulation_steps=2` |
| Gradient checkpointing | Enabled |
| Precision | bf16 |
| Device | Apple Silicon (MPS) |
| Learning rate | 3e-5 |
| Weight decay | 0.01 |
| Max length | 512 tokens (source p95 ≈ 499) |
| Seed | 42 |

The small batch size and gradient checkpointing are MPS accommodations — on Apple
Silicon, GPU and host memory are one unified pool — not modeling choices.

`max_length=512` was measured, not inherited from BART's 1024-token ceiling: source
documents mean ≈172 tokens, p95 ≈499, p99 ≈742; targets mean ≈98. 512 covers ~p95 while
cutting self-attention's O(n²) memory cost to a quarter of the full ceiling.

Observed training loss (step | train | validation): 1250 | 0.7859 | 0.1956 —
2500 | 0.3665 | 0.1907.

## Status: read this before citing it

**No document-level evaluation result exists for this model.**

The correct benchmark — D-Wikipedia's own test split scored with D-SARI, LENS and
BERTScore — is implemented in the project but has not been run against this checkpoint.

What *was* run is a mismatched evaluation, reported here only for honesty about what
exists. This checkpoint was scored on **ASSET** (359 isolated *sentences*), which is the
wrong benchmark for a document-trained model:

| | SARI ↑ | BLEU | FKGL ↓ |
|---|---|---|---|
| This model (on ASSET — **wrong benchmark**) | 33.54 | 50.83 | 7.71 |
| `facebook/bart-base` zero-shot | 21.34 | 89.89 | 10.02 |

The direction is encouraging (SARI up, FKGL down, BLEU down — the same pattern as the
sentence-level model) and suggests some transfer from document-level training to
sentence-level output. **It is not a measurement of document-level simplification
quality** and must not be quoted as one.

## Limitations

- **Preliminary and reduced-scope.** A full-corpus run has not completed.
- **Unevaluated on its actual task**, as above.
- **Output is not aligned to input.** By design it deletes, merges and reorders — you
  cannot map a sentence in the output back to a sentence in the input. Any UI must
  replace a whole block, not individual paragraphs.
- **Requires corpus-style input and output post-processing** (see above). This is not
  cosmetic; skipping it caused a measured factual hallucination.
- **Silent truncation** past 512 tokens.
- **No human evaluation**, and only an unexamined qualitative spot-check on a handful
  of real test-split documents.
- **English only**, encyclopedic register.

## Training data provenance and licence

D-Wikipedia is derived from English Wikipedia and Simple English Wikipedia (**CC BY-SA**).
This model is released under **CC BY-SA 4.0** accordingly.

> Sun, R., Jin, H., & Wan, X. (2021). *Document-Level Text Simplification: Dataset,
> Criteria and Baseline.* EMNLP 2021.

## Citation

Produced for a bachelor thesis (2026). Please cite the project repository:
<https://github.com/yyvs/simple-website>