File size: 1,066 Bytes
3551500 0c3e462 3551500 0c3e462 3551500 0c3e462 3551500 0c3e462 3551500 0c3e462 3551500 0c3e462 3551500 0c3e462 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 |
---
language:
- en
license: openrail
tags:
- diffusion-llm
- parallel-generation
- custom-transformer
- cropmark
datasets:
- OpenAssistant/oasst1
metrics:
- cosine_similarity
base_model:
- darwinkernelpanic/DiffReaper-5
---
# DiffReaper-5L
DiffReaper-5L is a **larger** version of DiffReaper-5, with **2048-dim embeddings** and a **24-layer Transformer**.
## Model Details
- **Architecture:** 24-layer Custom Transformer with Time Embedding.
- **Task:** Conditioned Text Diffusion (Prompt-Response).
- **Training Objective:** Cosine Similarity Regression.
- **Sampling:** 10-step iterative parallel denoising.
## Usage (Inference)
To run inference:
```python
import torch
# Assuming DiffReaperModel is defined as in train_diffreaper_5l.py
model = DiffReaperModel(vocab_size=50257, n_embd=2048, n_head=32, n_layer=24).to("cuda")
model.load_state_dict(torch.load("diffreaper5l_latest.pt"))
model.eval()
```
## Fine-tuning
To fine-tune on a custom dataset, ensure your data loader provides **Prompt** + **Response** pairs. Use the same Cosine Similarity loss. |