| --- |
| language: |
| - en |
| --- |
| # sd 1.5 vae, text trained demo |
|
|
| It has been said that the sd vae is not capable of faithful text rendering, |
| at a fundamental level. |
| I decided to test that and it turns out to not exactly be true. |
|
|
| This 4channel vae can render a 16pt font with almost perfect fidelity. |
|
|
| This is a TECH DEMO ONLY. I have trained it exclusively on text, so |
| normal rendering has drastically suffered as a result. |
|
|
| # Training details |
| This started with the original sd vae. |
| I generated an initial dataset of 14pt-only text images, with |
| https://github.com/ppbrown/ai-training/blob/main/trainer/vae/gen_glyph_dataset.py |
| ( around 4000 of them) |
| I then trained the vae at LR 1e-5 for 28000 steps, batchsize 1, |
| using my standard settings of lpips/vgg weights 0.08, LAP 0.02 |
| and "rawvgg" enabled, with my train_vae.py in that same repo |
| |
| I then generated a MIXED size font dataset,ranging across 14,16,18, and 20pt |
| total images around 8000 |
| This time I chose batch size 2, LR 2e-5. |
| |
| 14pt text is of course more difficult than 16pt text. |
| 16pt, however, renders almost indistinguishable from original. |
| The specific font used across all images, was |
| Libertinus Serif |
| |
| I did have a fairly good result at 69000 steps for 14pt. |
| I thought it had hit a floor, but then suddenly at 93000 steps, |
| there was a dramatic breakthrough, at least by the numbers. |
| (and then there was another one at 157000) |
| |
| !(clean 16pt)[bookimg_16.step_093000.webp] |
| |
| If you want to try it out for yourself, you can use the tools in my repo, |
| such as |
| |
| trainer/cache-utils/create_imgcache_sdvae.py --vae --previewonly \ |
| --model opendiffusionai/sd15vae-texttrained --data_root someimgdir |
