File size: 2,970 Bytes
de0dda2 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 | ---
license: cc-by-nc-4.0
pipeline_tag: text-to-image
tags:
- multimodal
- flow-matching
- image-text-to-text
- text-generation
datasets:
- benjamin-paine/imagenet-1k-256x256
language:
- en
base_model:
- UPShf/FlowTalk
---
# FlowTalk (Prototype Model Card)
## Summary
This is an experimental research prototype multimodal model that combines:
- Flow-matching image generation in VAE latent space
- Autoregressive text generation (next-token prediction)
It is not a production-quality text-to-image model. Prompt adherence is inconsistent and strongly depends on
matching the training prompt format used during training.
## Model Details
- Architecture: single multimodal transformer (see `omni_model_v2.py` in the code repository)
- Image path: predicts a flow/velocity in VAE latent space and decodes through a VAE
- Default VAE used by the scripts: `black-forest-labs/FLUX.1-schnell`
- Text path: next-token prediction head
This model is brittle under distribution shift and is best treated as a research artifact.
## Training Data
This checkpoint was trained on an ImageNet-derived 256x256 dataset hosted on Hugging Face:
- Dataset: `benjamin-paine/imagenet-1k-256x256`
- Dataset license field on HF: "other"
- ImageNet usage terms in the dataset card: non-commercial research / educational
Captions were generated with a VLM (Qwen-VL style captions), and some runs use ChatML-like prompt templates.
## Prompt Format Warning (Critical)
If your training captions were ChatML-ish (tokens like `<|im_start|>user`, `<|im_end|>`), then plain prompts like:
`green trees, flowers`
are out-of-distribution and may produce weak prompt control. For best results, use the same template used to
create training captions (or retrain using plain captions).
## Intended Use
- Research on flow-matching multimodal transformers
- Captioning / tagging experiments (quality depends heavily on training data)
- Debugging and ablation studies
## Limitations
- Not reliable for real-world prompt-following
- Can collapse to near-constant outputs (especially under prompt-format mismatch)
- Text generation quality is not competitive with production LLMs
- No safety mitigations; may generate unsafe content depending on training data
## How To Use
Code repository (scripts, not a library):
- https://github.com/uninterruptedpowersupply3-NEW/FlowTalk
Typical usage is via `gui_app.py` and `inference_backend.py` in the code repository.
## License
- Code: Apache-2.0 (see the code repository)
- Weights: CC BY-NC 4.0 (non-commercial)
This checkpoint was trained on ImageNet-derived data; users are responsible for complying with ImageNet terms.
## Citation
If you use this checkpoint or the codebase, please cite ImageNet and any upstream components you used (VAE,
captioning model, etc.).
## References
- Dataset: https://huggingface.co/datasets/benjamin-paine/imagenet-1k-256x256
- Default VAE repo: https://huggingface.co/black-forest-labs/FLUX.1-schnell |