README.md

---
license: apache-2.0
datasets:
- ibrahimhamamci/CT-RATE
- dmolino/CT-RATE_Generated_Scans
language:
- en
- tr
- it
pipeline_tag: text-to-3d
tags:
- medical
---
# Text-to-CT Model Weights

Checkpoints for **“Text-to-CT Generation via 3D Latent Diffusion Model with Contrastive Vision-Language Pretraining”** (Molino et al., 2025).
---

## Model Card for Text-to-CT Generation via 3D Latent Diffusion Model with Contrastive Vision-Language Pretraining

### Model Description
- **Authors:** Daniele Molino, Camillo Maria Caruso, Filippo Ruffini, Paolo Soda, Valerio Guarrasi  
- **Model type:** 3D latent diffusion (RFlow) + 3D VAE + CLIP3D text encoder for CT generation.  
- **License:** Apache 2.0 (same as code release).  
- **Sources:** Code https://github.com/cosbidev/Text2CT | Paper https://arxiv.org/abs/2506.00633  
- **Demo:** Use `diff_model_demo.py` from the code release for a one-off generation from text.

### Intended Use
- **Direct use:** Research/experimentation on text-conditioned 3D CT synthesis; generating synthetic data for benchmarking or augmentation.  
- **Downstream use:** Fine-tuning or integration into broader research pipelines.  
- **Out of scope:** Clinical decision-making, diagnostic use, or deployment without proper validation and approvals.

### Risks & Limitations
- Trained on CT-RATE; may encode dataset biases and is not validated for clinical use.
- Synthetic outputs may contain artifacts; do not use for patient care.

### Files included
- `autoencoder_epoch273.pt` — 3D VAE for latent compression/decoding.
- `unet_rflow_200ep.pt` — Diffusion UNet trained with rectified flow.
- `CLIP3D_Finding_Impression_30ep.pt` — CLIP3D weights for encoding reports.

### How to Get Started (Python)
```python
from huggingface_hub import snapshot_download

repo_id = "dmolino/text2ct-weights"

snapshot_download(
    repo_id=repo_id,
    repo_type="model",
    local_dir="your_local_path" 
)
```

# Use these in the code release configs:
# trained_autoencoder_path -> autoencoder_path
# existing_ckpt_filepath / model_filename -> unet_path
# clip_weights (for report embeddings) -> clip_path
```

### Training Data (for these weights)
- CT-RATE dataset (public on Hugging Face) for CT volumes and reports.

### Training Procedure (summary)
- CLIP3D trained for vision-language alignment on CT+reports.
- VAE checkpoint from https://github.com/Project-MONAI/tutorials/tree/main/generation/maisi.
- Diffusion UNet trained with rectified flow (RFlow) in latent space, conditioned on text embeddings.

### Evaluation
- See paper for quantitative and qualitative results.

### Further Information
- 1,000 generated CT scans are available at https://huggingface.co/datasets/dmolino/CT-RATE_Generated_Scans.

### Environmental Impact
- Not reported. Training used multi-GPU setup;.

### Citation
If you use these weights or code, please cite the paper:
```
@article{molino2025text,
  title={Text-to-CT Generation via 3D Latent Diffusion Model with Contrastive Vision-Language Pretraining},
  author={Molino, Daniele and Caruso, Camillo Maria and Ruffini, Filippo and Soda, Paolo and Guarrasi, Valerio},
  journal={arXiv preprint arXiv:2506.00633},
  year={2025}
}
```