| --- |
| license: apache-2.0 |
| base_model: unsloth/Meta-Llama-3.1-8B-Instruct |
| tags: |
| - diffusion |
| - language-model |
| - llama |
| - text-generation |
| library_name: transformers |
| pipeline_tag: text-generation |
| --- |
| |
| # Llama-3.1-8B Diffusion Model (LAD) |
|
|
| This is a **Language Autoregressive Diffusion (LAD)** model based on Llama-3.1-8B-Instruct. |
|
|
| ## Features |
| - 🎯 Dual mode: Autoregressive + Diffusion generation |
| - 🚀 Cosine noise schedule with 1000 timesteps |
| - 🧠 LoRA fine-tuning (rank 32) |
| - ⚡ Custom diffusion components |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoTokenizer, AutoModelForCausalLM |
| |
| model = AutoModelForCausalLM.from_pretrained("rootxhacker/llama3-diffusion") |
| tokenizer = AutoTokenizer.from_pretrained("rootxhacker/llama3-diffusion") |
| |
| # Generate text |
| inputs = tokenizer("The future of AI", return_tensors="pt") |
| outputs = model.generate(**inputs, max_length=100) |
| print(tokenizer.decode(outputs[0])) |
| ``` |
|
|
| ## Training Details |
| - Base: Meta-Llama-3.1-8B-Instruct |
| - Dataset: PatrickHaller/cosmopedia-v2-1B |
| - Framework: Unsloth + Custom Diffusion |
| - Context: 256 tokens |
| - Training: 60% AR + 40% Diffusion |
|
|
| Uploaded: 2025-06-08 23:13 |
|
|
