TextRIRO / README.md
Yesianrohn's picture
Create README.md
1f58d42 verified
|
Raw History Blame Contribute Delete
5.75 kB
---
language:
- en
pipeline_tag: image-to-image
base_model: stable-diffusion-v1-5/stable-diffusion-v1-5
datasets:
- topdu/OpenOCR-Data
tags:
- scene-text-editing
- image-editing
- conditional-diffusion
- pytorch
- arxiv:2610.27592
---
# TextRIRO
**Official model checkpoint for [Real In, Real Out: What If We Only Use Real Data for Scene Text Editing?](https://huggingface.co/papers/2610.27592) (NeurIPS 2026).**
Xingsong Ye, Yongkun Du, Jiaxin Zhang, Chong Sun, and Zhineng Chen
[HF Paper](https://huggingface.co/papers/2610.27592) 路 [arXiv](https://arxiv.org/abs/2610.27592) 路 [NeurIPS 2026](https://neurips.cc/virtual/2026/loc/sydney/poster/153296) 路 [Code](https://github.com/YesianRohn/TextRIRO) 路 [Model](https://huggingface.co/Yesianrohn/TextRIRO) 路 [Demo](https://huggingface.co/spaces/Yesianrohn/TextRIRO-Demo) 路 [TextRIRO-3M](https://huggingface.co/datasets/Yesianrohn/TextRIRO-3M) 路 [RIROCommon](https://huggingface.co/datasets/Yesianrohn/RIROCommon)
## Overview
TextRIRO uses conditional diffusion to replace text in a scene-text crop while preserving the reference appearance. **Shuffle&Crop** creates self-paired supervision by disrupting text semantics in real images while retaining style cues. A multi-scale style encoder and an STR-pretrained text encoder condition generation on appearance and target characters. Training requires no synthetic before-and-after editing pairs or character segmentation.
Here, **real-only training** refers to editing supervision and STR-based text-encoder pre-training. The generative backbone uses pretrained Stable Diffusion components.
![TextRIRO framework](https://raw.githubusercontent.com/YesianRohn/TextRIRO/main/assets/framework.png)
## Checkpoint and architecture
| Item | Details |
| --- | --- |
| Released weights | [`textriro.pth`](https://huggingface.co/Yesianrohn/TextRIRO/blob/main/textriro.pth) |
| Inputs | A cropped scene-text image and the desired replacement text |
| Output | An edited crop with the requested text |
| Backbone components | Stable Diffusion 1.5 U-Net and scheduler; Stable Diffusion 2.1 VAE |
| Text conditioning | STR-pretrained character embeddings provided in the code repository |
| Generation resolution | 64 脳 256 pixels; the inference script restores the source dimensions |
| Training data | U14M-Filter, the filtered real-image subset of Union14M, distributed through [OpenOCR-Data](https://huggingface.co/datasets/topdu/OpenOCR-Data/tree/main/Union14M-L-LMDB-Filtered) |
This repository distributes a PyTorch checkpoint for the project's custom implementation. Use the loading and inference scripts in the [code repository](https://github.com/YesianRohn/TextRIRO).
## Quick start
### 1. Prepare the implementation
```bash
git clone https://github.com/YesianRohn/TextRIRO.git
cd TextRIRO
```
Follow the repository's [installation and checkpoint preparation instructions](https://github.com/YesianRohn/TextRIRO#installation). The inference script requires CUDA. Prepare the SD 1.5 U-Net/scheduler, SD 2.1 VAE, and STR-pretrained text encoder at the paths specified in [`configs/train.yaml`](https://github.com/YesianRohn/TextRIRO/blob/main/configs/train.yaml).
### 2. Download the checkpoint
Run this from the cloned repository root:
```python
from huggingface_hub import hf_hub_download
hf_hub_download(
repo_id="Yesianrohn/TextRIRO",
filename="textriro.pth",
local_dir=".",
)
```
### 3. Prepare images and replacement text
Put source crops in `example/i_s/`. Create `example/i_t.txt` with one image filename and replacement string per line:
```text
sample_001.png FLASH
sample_002.jpg Real In Real Out
```
Separate the filename and target with one space. Spaces inside the target are supported. Every source image must have a matching entry.
### 4. Generate edited crops
```bash
python inference.py \
--ckpt_path textriro.pth \
--dataset_dir example \
--output_dir example_result \
--num_inference_steps 20 \
--guidance_scale 2.2 \
--seed 42
```
Outputs are written to `example_result/` using the source filenames. To try the model interactively, visit the [online demo](https://huggingface.co/spaces/Yesianrohn/TextRIRO-Demo).
## Scope and limitations
- The released inference script processes text crops. Full-image text detection and compositing require additional processing.
- The current encoder accepts up to **25 target characters**; longer strings are truncated.
- Supported characters are defined by [`EN_symbol_dict.txt`](https://github.com/YesianRohn/TextRIRO/blob/main/EN_symbol_dict.txt) plus the space character. The release should not be treated as a general multilingual editor.
- Output text accuracy and visual consistency should be checked for each application. Detailed evaluations and comparisons are available in the [paper](https://huggingface.co/papers/2610.27592).
## Related resources
- [TextRIRO-3M](https://huggingface.co/datasets/Yesianrohn/TextRIRO-3M): a synthetic long-text recognition dataset created with TextRIRO.
- [RIROCommon](https://huggingface.co/datasets/Yesianrohn/RIROCommon): a benchmark covering equal-length and variable-length scene text editing.
- [Source code](https://github.com/YesianRohn/TextRIRO): training, inference, configurations, and evaluation guidance.
## Citation
If you use TextRIRO in your research, please cite:
```bibtex
@inproceedings{ye2026textriro,
title = {Real In, Real Out: What If We Only Use Real Data for Scene Text Editing?},
author = {Ye, Xingsong and Du, Yongkun and Zhang, Jiaxin and Sun, Chong and Chen, Zhineng},
booktitle = {Advances in Neural Information Processing Systems},
year = {2026},
url = {https://neurips.cc/virtual/2026/loc/sydney/poster/153296}
}
```