---
language:
- en
license: openrail
tags:
- diffusion-llm
- parallel-generation
- custom-transformer
- cropmark
datasets:
- OpenAssistant/oasst1
metrics:
- cosine_similarity
base_model:
- darwinkernelpanic/DiffReaper-5
---

# DiffReaper-5L

DiffReaper-5L is a **larger** version of DiffReaper-5, with **2048-dim embeddings** and a **24-layer Transformer**.

## Model Details

- **Architecture:** 24-layer Custom Transformer with Time Embedding.
- **Task:** Conditioned Text Diffusion (Prompt-Response).
- **Training Objective:** Cosine Similarity Regression.
- **Sampling:** 10-step iterative parallel denoising.

## Usage (Inference)

To run inference:

```python
import torch
# Assuming DiffReaperModel is defined as in train_diffreaper_5l.py

model = DiffReaperModel(vocab_size=50257, n_embd=2048, n_head=32, n_layer=24).to("cuda")
model.load_state_dict(torch.load("diffreaper5l_latest.pt"))
model.eval()
```

## Fine-tuning

To fine-tune on a custom dataset, ensure your data loader provides **Prompt** + **Response** pairs. Use the same Cosine Similarity loss.