---
license: other
tags:
  - non-commercial
  - text-generation
  - flow-matching
datasets:
  - cerebras/SlimPajama-627B
---

# DFM

## Summary
`DFM` is a continued-pretraining checkpoint based on Apple's fs-dfm weights. It is trained with Flow Matching code and released for research/non-commercial use only.
This model was continued from a uniform‑noise trained checkpoint to a masked‑diffusion variant.
Base checkpoint (external, not on HF):
```
https://ml-site.cdn-apple.com/models/fs-dfm/checkpoint.pth
```

## Training
- Continued pretraining from Apple's fs-dfm checkpoint. Init: uniform‑noise checkpoint → continued training to mask‑diffusion
- Dataset: SlimPajama-627B
- Steps: 250,000
- Global batch size: 256

## License
Research/non-commercial use only. This repository is governed by the Apple Software License (see `LICENSE`) and includes non-commercial restrictions inherited from Flow Matching (CC BY-NC 4.0). See `ACKNOWLEDGMENTS` for third-party notices.

## Intended Use
Research and non-commercial use only.

## Limitations
Commercial use is not permitted. Dataset-specific licensing constraints apply to SlimPajama's underlying sources.

## Usage
### Hugging Face (trust_remote_code)
This repo provides `configuration_dfm.py` and `modeling_dfm.py` for HF loading with `trust_remote_code=True`.

Example:
```python
from transformers import AutoConfig, AutoModel

config = AutoConfig.from_pretrained(".", trust_remote_code=True)
model = AutoModel.from_pretrained(".", trust_remote_code=True)
```

Note:
- This model expects `x_t` and `time` inputs (flow-matching style), not GPT-style autoregressive inputs.

This release includes model-only weights (`model.safetensors`) for inference/forward passes. Full training/eval/sampling code is available in the original project: `https://github.com/apple/ml-fs-dfm`.

## Acknowledgments
This model is derived from Apple's fs-dfm checkpoint and follows the original Apple license terms. The original project is at `https://github.com/apple/ml-fs-dfm`. See `ACKNOWLEDGMENTS` for third-party attributions and licensing.