---
license: mit
---

# Custom Llama-style 

This repository contains a single `.pt` checkpoint file from a fine-tuned model.

**This model is NOT directly usable with `transformers.AutoModel.from_pretrained()` yet.** It needs to be converted to the Hugging Face format first.

## Training Details

- **Framework:**  [modded-nanoGPT-soap](https://github.com/nikhilvyas/modded-nanogpt-SOAP).
- **Architecture:** This model uses modern features and is NOT a standard GPT-2.
  - **Positional Embeddings:** Rotary Position Embeddings (RoPE)
  - **Normalization:** RMSNorm
  - **Bias:** Linear layers trained with `bias=False`.

## Model Configuration

This is the information needed to perform the conversion:

- `n_layer`: 12  
- `n_head`: 12  
- `n_embd`: 768 
- `vocab_size`: 50257 
- `block_size`: 1024

## Tokenizer

The model was trained with the standard `gpt2` tokenizer.