File size: 2,090 Bytes
7b6c98e 4641771 7b6c98e 4641771 7b6c98e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 | ---
license: other
tags:
- non-commercial
- text-generation
- flow-matching
datasets:
- cerebras/SlimPajama-627B
---
# DFM
## Summary
`DFM` is a continued-pretraining checkpoint based on Apple's fs-dfm weights. It is trained with Flow Matching code and released for research/non-commercial use only.
This model was continued from a uniform‑noise trained checkpoint to a masked‑diffusion variant.
Base checkpoint (external, not on HF):
```
https://ml-site.cdn-apple.com/models/fs-dfm/checkpoint.pth
```
## Training
- Continued pretraining from Apple's fs-dfm checkpoint. Init: uniform‑noise checkpoint → continued training to mask‑diffusion
- Dataset: SlimPajama-627B
- Steps: 250,000
- Global batch size: 256
## License
Research/non-commercial use only. This repository is governed by the Apple Software License (see `LICENSE`) and includes non-commercial restrictions inherited from Flow Matching (CC BY-NC 4.0). See `ACKNOWLEDGMENTS` for third-party notices.
## Intended Use
Research and non-commercial use only.
## Limitations
Commercial use is not permitted. Dataset-specific licensing constraints apply to SlimPajama's underlying sources.
## Usage
### Hugging Face (trust_remote_code)
This repo provides `configuration_dfm.py` and `modeling_dfm.py` for HF loading with `trust_remote_code=True`.
Example:
```python
from transformers import AutoConfig, AutoModel
config = AutoConfig.from_pretrained(".", trust_remote_code=True)
model = AutoModel.from_pretrained(".", trust_remote_code=True)
```
Note:
- This model expects `x_t` and `time` inputs (flow-matching style), not GPT-style autoregressive inputs.
This release includes model-only weights (`model.safetensors`) for inference/forward passes. Full training/eval/sampling code is available in the original project: `https://github.com/apple/ml-fs-dfm`.
## Acknowledgments
This model is derived from Apple's fs-dfm checkpoint and follows the original Apple license terms. The original project is at `https://github.com/apple/ml-fs-dfm`. See `ACKNOWLEDGMENTS` for third-party attributions and licensing.
|