README.md

metadata

license: cc-by-nc-4.0
pipeline_tag: text-to-image
tags:
  - multimodal
  - flow-matching
  - image-text-to-text
  - text-generation
datasets:
  - benjamin-paine/imagenet-1k-256x256
language:
  - en
base_model:
  - UPShf/FlowTalk

FlowTalk (Prototype Model Card)

Summary

This is an experimental research prototype multimodal model that combines:

• Flow-matching image generation in VAE latent space
• Autoregressive text generation (next-token prediction)

It is not a production-quality text-to-image model. Prompt adherence is inconsistent and strongly depends on matching the training prompt format used during training.

Model Details

• Architecture: single multimodal transformer (see omni_model_v2.py in the code repository)
• Image path: predicts a flow/velocity in VAE latent space and decodes through a VAE
  • Default VAE used by the scripts: black-forest-labs/FLUX.1-schnell
• Text path: next-token prediction head

This model is brittle under distribution shift and is best treated as a research artifact.

Training Data

This checkpoint was trained on an ImageNet-derived 256x256 dataset hosted on Hugging Face:

• Dataset: benjamin-paine/imagenet-1k-256x256
• Dataset license field on HF: "other"
• ImageNet usage terms in the dataset card: non-commercial research / educational

Captions were generated with a VLM (Qwen-VL style captions), and some runs use ChatML-like prompt templates.

Prompt Format Warning (Critical)

If your training captions were ChatML-ish (tokens like <|im_start|>user, <|im_end|>), then plain prompts like:

green trees, flowers

are out-of-distribution and may produce weak prompt control. For best results, use the same template used to create training captions (or retrain using plain captions).

Intended Use

• Research on flow-matching multimodal transformers
• Captioning / tagging experiments (quality depends heavily on training data)
• Debugging and ablation studies

Limitations

• Not reliable for real-world prompt-following
• Can collapse to near-constant outputs (especially under prompt-format mismatch)
• Text generation quality is not competitive with production LLMs
• No safety mitigations; may generate unsafe content depending on training data

How To Use

Code repository (scripts, not a library):

• https://github.com/uninterruptedpowersupply3-NEW/FlowTalk

Typical usage is via gui_app.py and inference_backend.py in the code repository.

License

• Code: Apache-2.0 (see the code repository)
• Weights: CC BY-NC 4.0 (non-commercial)

This checkpoint was trained on ImageNet-derived data; users are responsible for complying with ImageNet terms.

Citation

If you use this checkpoint or the codebase, please cite ImageNet and any upstream components you used (VAE, captioning model, etc.).

References

• Dataset: https://huggingface.co/datasets/benjamin-paine/imagenet-1k-256x256
• Default VAE repo: https://huggingface.co/black-forest-labs/FLUX.1-schnell