--- license: cc-by-sa-4.0 language: - en base_model: facebook/bart-base # See the sentence model's card for why pipeline_tag is left unset: the Hub rejects # "text2text-generation" and mislabels the model as decoder-only without it. tags: - text-simplification - document-simplification - accessibility - bart datasets: - d-wikipedia --- # bart-base-dwikipedia-simplification `facebook/bart-base` fine-tuned on **D-Wikipedia** (Sun et al., EMNLP 2021) for **document-level** English text simplification. Unlike a sentence-level simplifier, it rewrites a whole multi-sentence passage at once and is *expected* to delete, merge, split and reorder sentences — so its output has **no position-by-position correspondence to its input**. Trained as part of a bachelor thesis on automated simplification of everyday English web text, and served as the `document` model by the project's backend: > ⚠️ **Preliminary checkpoint.** This is a *reduced-scope* run (20,000-document train > slice, 2 epochs), and it has **never been evaluated on a document-level benchmark**. > See [Status](#status-read-this-before-citing-it) before using or citing it. ## ⚠️ Input and output are lowercased and PTB-pre-tokenized This is the most important usage detail, and getting it wrong degrades output badly. D-Wikipedia is fully lowercased, PTB-pre-tokenized (`writer , intellectual`, `women 's`, ``` `` the second sex '' ```) and one document per line, with **zero newline characters inside any document body** (verified across 3,000 documents). The model both **consumes and emits** that convention. Measured consequences of ignoring it: - Feeding ordinary mixed-case prose produced a **factual hallucination** ("northern Netherlands" → "northern hemisphere") that the lowercased, corpus-style input did not. - Supplying `\n` as a structural separator — the intuitive way to convey document structure — produced the **worst output of every variant tested**, losing more content and hallucinating more than either alternative, because the model has never seen a newline in training. Raw output looks like `achtkarspelen is a municipality in friesland .` — you must de-normalize it before display. The companion project implements the matched normalize/de-normalize pair in [`backend/document_text.py`](https://github.com/yyvs/simple-website/blob/main/backend/document_text.py): it re-attaches punctuation, restores quotes, then recovers casing in two passes (proper nouns and acronyms recovered from the *source* text, since the model only ever emits lowercase, followed by sentence-initial capitals). ## Usage ```python from transformers import AutoTokenizer, AutoModelForSeq2SeqLM tok = AutoTokenizer.from_pretrained("yunvs/bart-base-dwikipedia-simplification") model = AutoModelForSeq2SeqLM.from_pretrained("yunvs/bart-base-dwikipedia-simplification") # Normalize to corpus style first: lowercase, PTB-tokenize, no newlines. doc = "achtkarspelen is a municipality in the northern netherlands . it had a population of 27,944 in 2017 ." inputs = tok(doc, return_tensors="pt", truncation=True, max_length=512) out = model.generate(**inputs, max_length=512, num_beams=4, no_repeat_ngram_size=3, repetition_penalty=1.2) print(tok.decode(out[0], skip_special_tokens=True)) # -> lowercase, pre-tokenized output; de-normalize before displaying ``` **Input past 512 tokens is truncated, not chunked** — the overflow is silently dropped rather than simplified. Chunk below the limit yourself. A real Wikipedia article measured 2,459 BART tokens, i.e. ~79% would have been discarded if fed whole. **Do not pack input to fill 512.** The training median is 120 tokens and 68% of training documents are ≤200, while only 8% are ≥450. A lone 100-token section is *more* representative of the training distribution than a merged 500-token one — 512 is a ceiling for splitting, never a target for packing. **Do not include headings.** When a heading was included in a section's text, the model **echoed it back into the body output**. Headings are also far shorter than anything it saw in training. ## Training | | | |---|---| | Base model | `facebook/bart-base` | | Dataset | D-Wikipedia (Sun et al., EMNLP 2021), 20,000-document train slice | | Scope | **"reduced"** — not a full-corpus run | | Epochs | 2 | | Batch size | 2, with `gradient_accumulation_steps=2` | | Gradient checkpointing | Enabled | | Precision | bf16 | | Device | Apple Silicon (MPS) | | Learning rate | 3e-5 | | Weight decay | 0.01 | | Max length | 512 tokens (source p95 ≈ 499) | | Seed | 42 | The small batch size and gradient checkpointing are MPS accommodations — on Apple Silicon, GPU and host memory are one unified pool — not modeling choices. `max_length=512` was measured, not inherited from BART's 1024-token ceiling: source documents mean ≈172 tokens, p95 ≈499, p99 ≈742; targets mean ≈98. 512 covers ~p95 while cutting self-attention's O(n²) memory cost to a quarter of the full ceiling. Observed training loss (step | train | validation): 1250 | 0.7859 | 0.1956 — 2500 | 0.3665 | 0.1907. ## Status: read this before citing it **No document-level evaluation result exists for this model.** The correct benchmark — D-Wikipedia's own test split scored with D-SARI, LENS and BERTScore — is implemented in the project but has not been run against this checkpoint. What *was* run is a mismatched evaluation, reported here only for honesty about what exists. This checkpoint was scored on **ASSET** (359 isolated *sentences*), which is the wrong benchmark for a document-trained model: | | SARI ↑ | BLEU | FKGL ↓ | |---|---|---|---| | This model (on ASSET — **wrong benchmark**) | 33.54 | 50.83 | 7.71 | | `facebook/bart-base` zero-shot | 21.34 | 89.89 | 10.02 | The direction is encouraging (SARI up, FKGL down, BLEU down — the same pattern as the sentence-level model) and suggests some transfer from document-level training to sentence-level output. **It is not a measurement of document-level simplification quality** and must not be quoted as one. ## Limitations - **Preliminary and reduced-scope.** A full-corpus run has not completed. - **Unevaluated on its actual task**, as above. - **Output is not aligned to input.** By design it deletes, merges and reorders — you cannot map a sentence in the output back to a sentence in the input. Any UI must replace a whole block, not individual paragraphs. - **Requires corpus-style input and output post-processing** (see above). This is not cosmetic; skipping it caused a measured factual hallucination. - **Silent truncation** past 512 tokens. - **No human evaluation**, and only an unexamined qualitative spot-check on a handful of real test-split documents. - **English only**, encyclopedic register. ## Training data provenance and licence D-Wikipedia is derived from English Wikipedia and Simple English Wikipedia (**CC BY-SA**). This model is released under **CC BY-SA 4.0** accordingly. > Sun, R., Jin, H., & Wan, X. (2021). *Document-Level Text Simplification: Dataset, > Criteria and Baseline.* EMNLP 2021. ## Citation Produced for a bachelor thesis (2026). Please cite the project repository: