---
license: cc-by-nc-4.0
pipeline_tag: text-to-image
tags:
- multimodal
- flow-matching
- image-text-to-text
- text-generation
datasets:
- benjamin-paine/imagenet-1k-256x256
language:
- en
base_model:
- UPShf/FlowTalk
---


# FlowTalk (Prototype Model Card)

## Summary

This is an experimental research prototype multimodal model that combines:

- Flow-matching image generation in VAE latent space
- Autoregressive text generation (next-token prediction)

It is not a production-quality text-to-image model. Prompt adherence is inconsistent and strongly depends on
matching the training prompt format used during training.

## Model Details

- Architecture: single multimodal transformer (see `omni_model_v2.py` in the code repository)
- Image path: predicts a flow/velocity in VAE latent space and decodes through a VAE
  - Default VAE used by the scripts: `black-forest-labs/FLUX.1-schnell`
- Text path: next-token prediction head

This model is brittle under distribution shift and is best treated as a research artifact.

## Training Data

This checkpoint was trained on an ImageNet-derived 256x256 dataset hosted on Hugging Face:

- Dataset: `benjamin-paine/imagenet-1k-256x256`
- Dataset license field on HF: "other"
- ImageNet usage terms in the dataset card: non-commercial research / educational

Captions were generated with a VLM (Qwen-VL style captions), and some runs use ChatML-like prompt templates.

## Prompt Format Warning (Critical)

If your training captions were ChatML-ish (tokens like `<|im_start|>user`, `<|im_end|>`), then plain prompts like:

`green trees, flowers`

are out-of-distribution and may produce weak prompt control. For best results, use the same template used to
create training captions (or retrain using plain captions).

## Intended Use

- Research on flow-matching multimodal transformers
- Captioning / tagging experiments (quality depends heavily on training data)
- Debugging and ablation studies

## Limitations

- Not reliable for real-world prompt-following
- Can collapse to near-constant outputs (especially under prompt-format mismatch)
- Text generation quality is not competitive with production LLMs
- No safety mitigations; may generate unsafe content depending on training data

## How To Use

Code repository (scripts, not a library):

- https://github.com/uninterruptedpowersupply3-NEW/FlowTalk

Typical usage is via `gui_app.py` and `inference_backend.py` in the code repository.

## License

- Code: Apache-2.0 (see the code repository)
- Weights: CC BY-NC 4.0 (non-commercial)

This checkpoint was trained on ImageNet-derived data; users are responsible for complying with ImageNet terms.

## Citation

If you use this checkpoint or the codebase, please cite ImageNet and any upstream components you used (VAE,
captioning model, etc.).

## References

- Dataset: https://huggingface.co/datasets/benjamin-paine/imagenet-1k-256x256
- Default VAE repo: https://huggingface.co/black-forest-labs/FLUX.1-schnell