fmorgens/flux2-1d-tokenizer-preview

FLUX.2 1D Image Tokenizer — preview

Unofficial research / preview release. A 1D image tokenizer that compresses a frozen FLUX.2 VAE latent into 1024 discrete tokens × 16 bits (~2 KB/image), for use as a compact visual vocabulary in downstream autoregressive models. Ships two decoder variants that emit identical tokens.

⚠️ Not state of the art. Reconstruction quality sits around the "barely usable" line (see Limitations). Treat this as a compact, multi-resolution discrete tokenizer / research artifact — not a high-fidelity codec.

Use it

pip install flux2-1d-tokenizer

from flux2_tokenizer import Flux2Tokenizer

tok = Flux2Tokenizer.from_pretrained("fmorgens/flux2-1d-tokenizer-preview")                  # sharp
# tok = Flux2Tokenizer.from_pretrained("fmorgens/flux2-1d-tokenizer-preview", variant="base")

tokens = tok.encode(images)            # (B,3,H,W) in [-1,1]  ->  (B,1024) int64, 0..65535
recon  = tok.decode(tokens, 512, 512)  # (B,k<=1024)          ->  pixels in [-1,1]

Library + docs: https://github.com/fmorgens/flux2-1d-tokenizer

Examples

Reconstruction at 512² — columns are ground truth · base · sharp; rows are astronaut, coffee, chelsea, rocket. (256² and 768² versions are in samples/.)

Tail-drop — keeping only the first k tokens (columns: GT, then k = 8, 16, 32, 64, 128, 256, 512, 1024). For each image the top row is base, the bottom row sharp:

Variants (same tokens)

file	decoder	notes
sharp.safetensors (default)	adversarially fine-tuned (latent PatchGAN)	renders sharper; commits to plausible detail at low bit-rates instead of blurring. Detail is plausible, not guaranteed faithful (esp. text/faces)
base.safetensors	latent-MSE only	faithful/mean, softer

The encoder + codebook are frozen between them, so the token ids are byte-identical — encode once, decode with either. Each file is ~340 MB (fp32; tokenizer only — no VAE weights).

What it is

• Rate: 1024 tokens/image × 16 bits (65,536-entry codebook), independent of resolution.
• Resolutions: width & height each in {256, 512, 768}, any combination incl. rectangular; multiples of 16.
• Tail-drop: tokens are coarse-to-fine; keep the first k for a lower rate.
• Architecture: ~85 M-param transformer (encoder + VQ + decoder) operating in FLUX.2 VAE latent space (f8, 32-channel); normalized 2D RoPE. Vector quantizer with a 65,536-entry codebook of 128-d codes → 16 bits/token (the code_dim of 128 is the code-vector width, not the bit-rate). Not BSQ.
• Trained on ~1.2 M images, including ~400 k from LAION-Art. No ImageNet, LPIPS, or pretrained perceptual net — clean lineage.

Cross-resolution decode (a quirk)

Token identity is resolution-relative (normalized 2D RoPE), so the same tokens can be decoded at a different resolution or aspect ratio than the source — they encode a relative layout, not absolute pixels:

tokens = tok.encode(x_512)         # tokens from a 512x512 image
tok.decode(tokens, 768, 768)       # re-render at 768x768
tok.decode(tokens, 768, 256)       # ...or a different aspect ratio

This is an implicit side-effect of the resolution-relative positional encoding (the normalized 2D RoPE) — not a trained capability. The model was only ever trained to decode at the same resolution it encoded; it was never trained to resize. So cross-resolution decode is emergent and unsupervised — a curiosity that shows the tokens are layout-relative, not a reliable resampler.

The house image encoded at three sizes (256², 768², and an extreme 256×768), each decoded into all 9 resolutions (the tokenizer row), with a naive bilinear stretch of the source underneath (the stretch row) for comparison. Columns are the 9 target sizes (labeled on the grid).

Limitations

• Rate-capped. Faces, small text (only large glyphs, ~>32 px, survive), and fine high-frequency texture are soft. Flat / low-texture regions reconstruct well.
• sharp invents plausible detail at low k — good for perceptual sharpness, not for pixel-faithful or OCR use.
• Not benchmarked against SOTA tokenizers; expect worse rFID than dedicated codecs at this rate. This is an early release.

Requirements

Requires the FLUX.2 VAE (black-forest-labs/FLUX.2-VAE, Apache-2.0), downloaded at runtime — not redistributed here.

License & attribution

• Weights and the accompanying library are Apache-2.0.
• Built on the FLUX.2 VAE by Black Forest Labs (Apache-2.0).
• Sample images in this card are freely usable: a CC0 cat, two Unsplash-licensed photos (a house and a rice farmer), and an image from the Kodak test set.
• Unofficial. Not affiliated with or endorsed by Black Forest Labs. FLUX and FLUX.2 are trademarks of Black Forest Labs.