Blame for README.md

---
license: apache-2.0
base_model: OpenMOSS-Team/MOSS-Image-Base
base_model_relation: finetune
pipeline_tag: text-to-image
language:
  - sv
tags:
  - moss-image
  - handwriting
  - historical-documents
  - archives
  - htr
  - synthetic-data
---

# diffu-moss-r6b

A full fine-tune of the DiT of [MOSS-Image-Base](https://huggingface.co/OpenMOSS-Team/MOSS-Image-Base) that draws
whole synthetic archive pages: Swedish handwritten records of the 17th to 19th centuries. It is the drawing model of
[diffu](https://github.com/Borg93/diffu2), which makes training pages for page-level handwriting recognition: a page
layout (a PAGE/ALTO file, or the lines' boxes as JSON), the text to write in it (or the layout's own) and a sample page
go in; a page image and its exact ALTO and PAGE labels come out.

The model reads a MOSS Layout prompt, one text item per line with its exact text and its box, and a sample page given
as extra image tokens, whose paper, ink and look it follows. Only the DiT is in this repository. The text encoder,
tokenizer, VAE and scheduler are MOSS-Image-Base's, unchanged.

## Files

| File | What |
|---|---|
| `diffusion_pytorch_model.safetensors` | the DiT's weights, bfloat16 (10.2 GB, 462 tensors, the same names and shapes as Base's DiT) |
| `config.json` | `diffu_moss.transformer.MossTransformer` (diffu's own implementation of MOSS-Image's DiT) |
| `export.json` | `takes_reference: true`: the model takes a sample page |

## Use

With diffu (see its `docs/INFERENCE.md` for running on a Ray or KubeRay cluster):

```bash
hf download OpenMOSS-Team/MOSS-Image-Base --revision 85975793d8b3d432f1b6e27f352abaa4c2a4c0d3 --local-dir base
hf download Gabriel/diffu-moss-r6b --local-dir dit
git clone https://github.com/Borg93/diffu2 && cd diffu2 && uv sync --all-packages
uv run diffu-moss-init --base ../base --transformer ../dit --out ../moss-r6b
uv run diffu-run --jobs run.jsonl --draw-settings draw.json --model ../moss-r6b --out out --gpus 1
```

`run.jsonl` holds one page job per line, as JSON: the layout (`template`, a PAGE/ALTO file, or `layout`, the lines'
boxes in pixels), the text (left out, the layout's own), the style, the sample page and an id. diffu's `README.md` shows
two lines; `docs/INFERENCE.md` section 5 describes every field, with a worked example of each combination.

On Blackwell GPUs the DiT can run in MXFP8, quantized from these weights as the model loads: give the draw settings
`{"precision": "mxfp8", "compile_blocks": true}`. On a GB10 at 1024 x 1024 this draws at 1.38 s a step, against
1.76 s for bfloat16 compiled and 3.46 s uncompiled; its pages look like bfloat16's, slightly less saturated.

For SGLang Diffusion, `diffu-moss-export-hf` writes this DiT in the official MOSS-Image layout, and diffu's plugin
`diffu_sglang_moss` adds the sample page (`docs/INFERENCE.md` section 13).

Recommended drawing: about 1 MP, 50 steps, guidance 3, shift 5, block 22 skipped in the unconditional pass (MOSS-Image's
own recipe).

## Training

- Data: 3,406 page pairs balanced across eight collections of Swedish archival handwriting ground truth (court,
  chancery, mining and travel records): 2,031 whole pages and 1,375 windows of pages, about 1 MP each, each paired with
  a sample page from the same volume.
- Prompt forms: no sample, the volume named (20 %); the sample with the look in words (20 %); the sample with the
  volume named (20 %); the sample alone (40 %). In the last 1,000 steps half of the sample-alone pairs were restyled the
  same way on page and sample (microfilm grey, colourised, tinted), so the look has to come from the sample.
- Recipe: MOSS-Image's own full fine-tune: Muon on the joint blocks' matrices and AdamW elsewhere, learning rate 5e-5,
  weight decay 0.01, gradients clipped at 1, one pair a step, float32 weights under bfloat16 autocast; text encoder and
  VAE frozen.
- Steps: 1,500 from Base on the sample form, then 1,000 with the restyled pairs (2,500 in all), on one DGX Spark.

## Evaluation

On held-out pages of the training volumes, each drawn with its own sample and with samples of other volumes, from the
same noise:

| Check | Result |
|---|---|
| The page's own sample gives a page of the right kind (colour or grey) | 4 of 5 |
| A grey sample makes a colour page grey | 3 of 5 |
| Colour and grey changes together | 78 % |
| The sample's handwriting is followed instead of the volume's | 1 of 12 |

## Limitations

- The handwriting does not follow the sample page. Trained on eight volumes, the model recognises a volume from its
  text and layout and writes that volume's hand. Training on many more volumes is being tested.
- The look follows the sample only in part (above).
- Every line has to be checked: diffu reads each drawn line back and marks a line it cannot confirm as unclear in the
  labels.
- Only eight volumes' hands and layouts were trained; other kinds of pages are untested.

## Licence

Apache-2.0, as MOSS-Image-Base.