| --- |
| license: cc-by-nc-4.0 |
| pipeline_tag: text-to-image |
| tags: |
| - multimodal |
| - flow-matching |
| - image-text-to-text |
| - text-generation |
| datasets: |
| - benjamin-paine/imagenet-1k-256x256 |
| language: |
| - en |
| base_model: |
| - UPShf/FlowTalk |
| --- |
| |
|
|
| # FlowTalk (Prototype Model Card) |
|
|
| ## Summary |
|
|
| This is an experimental research prototype multimodal model that combines: |
|
|
| - Flow-matching image generation in VAE latent space |
| - Autoregressive text generation (next-token prediction) |
|
|
| It is not a production-quality text-to-image model. Prompt adherence is inconsistent and strongly depends on |
| matching the training prompt format used during training. |
|
|
| ## Model Details |
|
|
| - Architecture: single multimodal transformer (see `omni_model_v2.py` in the code repository) |
| - Image path: predicts a flow/velocity in VAE latent space and decodes through a VAE |
| - Default VAE used by the scripts: `black-forest-labs/FLUX.1-schnell` |
| - Text path: next-token prediction head |
|
|
| This model is brittle under distribution shift and is best treated as a research artifact. |
|
|
| ## Training Data |
|
|
| This checkpoint was trained on an ImageNet-derived 256x256 dataset hosted on Hugging Face: |
|
|
| - Dataset: `benjamin-paine/imagenet-1k-256x256` |
| - Dataset license field on HF: "other" |
| - ImageNet usage terms in the dataset card: non-commercial research / educational |
|
|
| Captions were generated with a VLM (Qwen-VL style captions), and some runs use ChatML-like prompt templates. |
|
|
| ## Prompt Format Warning (Critical) |
|
|
| If your training captions were ChatML-ish (tokens like `<|im_start|>user`, `<|im_end|>`), then plain prompts like: |
|
|
| `green trees, flowers` |
|
|
| are out-of-distribution and may produce weak prompt control. For best results, use the same template used to |
| create training captions (or retrain using plain captions). |
|
|
| ## Intended Use |
|
|
| - Research on flow-matching multimodal transformers |
| - Captioning / tagging experiments (quality depends heavily on training data) |
| - Debugging and ablation studies |
|
|
| ## Limitations |
|
|
| - Not reliable for real-world prompt-following |
| - Can collapse to near-constant outputs (especially under prompt-format mismatch) |
| - Text generation quality is not competitive with production LLMs |
| - No safety mitigations; may generate unsafe content depending on training data |
|
|
| ## How To Use |
|
|
| Code repository (scripts, not a library): |
|
|
| - https://github.com/uninterruptedpowersupply3-NEW/FlowTalk |
|
|
| Typical usage is via `gui_app.py` and `inference_backend.py` in the code repository. |
|
|
| ## License |
|
|
| - Code: Apache-2.0 (see the code repository) |
| - Weights: CC BY-NC 4.0 (non-commercial) |
|
|
| This checkpoint was trained on ImageNet-derived data; users are responsible for complying with ImageNet terms. |
|
|
| ## Citation |
|
|
| If you use this checkpoint or the codebase, please cite ImageNet and any upstream components you used (VAE, |
| captioning model, etc.). |
|
|
| ## References |
|
|
| - Dataset: https://huggingface.co/datasets/benjamin-paine/imagenet-1k-256x256 |
| - Default VAE repo: https://huggingface.co/black-forest-labs/FLUX.1-schnell |
