Update README.md

README.md

@@ -1,3 +1,94 @@
----
-license: cc-by-nc-4.0
----
+---
+license: cc-by-nc-4.0
+pipeline_tag: text-to-image
+tags:
+- multimodal
+- flow-matching
+- image-text-to-text
+- text-generation
+datasets:
+- benjamin-paine/imagenet-1k-256x256
+language:
+- en
+base_model:
+- UPShf/FlowTalk
+---
+
+
+# FlowTalk (Prototype Model Card)
+
+## Summary
+
+This is an experimental research prototype multimodal model that combines:
+
+- Flow-matching image generation in VAE latent space
+- Autoregressive text generation (next-token prediction)
+
+It is not a production-quality text-to-image model. Prompt adherence is inconsistent and strongly depends on
+matching the training prompt format used during training.
+
+## Model Details
+
+- Architecture: single multimodal transformer (see `omni_model_v2.py` in the code repository)
+- Image path: predicts a flow/velocity in VAE latent space and decodes through a VAE
+  - Default VAE used by the scripts: `black-forest-labs/FLUX.1-schnell`
+- Text path: next-token prediction head
+
+This model is brittle under distribution shift and is best treated as a research artifact.
+
+## Training Data
+
+This checkpoint was trained on an ImageNet-derived 256x256 dataset hosted on Hugging Face:
+
+- Dataset: `benjamin-paine/imagenet-1k-256x256`
+- Dataset license field on HF: "other"
+- ImageNet usage terms in the dataset card: non-commercial research / educational
+
+Captions were generated with a VLM (Qwen-VL style captions), and some runs use ChatML-like prompt templates.
+
+## Prompt Format Warning (Critical)
+
+If your training captions were ChatML-ish (tokens like `<|im_start|>user`, `<|im_end|>`), then plain prompts like:
+
+`green trees, flowers`
+
+are out-of-distribution and may produce weak prompt control. For best results, use the same template used to
+create training captions (or retrain using plain captions).
+
+## Intended Use
+
+- Research on flow-matching multimodal transformers
+- Captioning / tagging experiments (quality depends heavily on training data)
+- Debugging and ablation studies
+
+## Limitations
+
+- Not reliable for real-world prompt-following
+- Can collapse to near-constant outputs (especially under prompt-format mismatch)
+- Text generation quality is not competitive with production LLMs
+- No safety mitigations; may generate unsafe content depending on training data
+
+## How To Use
+
+Code repository (scripts, not a library):
+
+- https://github.com/uninterruptedpowersupply3-NEW/FlowTalk
+
+Typical usage is via `gui_app.py` and `inference_backend.py` in the code repository.
+
+## License
+
+- Code: Apache-2.0 (see the code repository)
+- Weights: CC BY-NC 4.0 (non-commercial)
+
+This checkpoint was trained on ImageNet-derived data; users are responsible for complying with ImageNet terms.
+
+## Citation
+
+If you use this checkpoint or the codebase, please cite ImageNet and any upstream components you used (VAE,
+captioning model, etc.).
+
+## References
+
+- Dataset: https://huggingface.co/datasets/benjamin-paine/imagenet-1k-256x256
+- Default VAE repo: https://huggingface.co/black-forest-labs/FLUX.1-schnell