README.md

---
base_model: mistralai/Mistral-Nemo-Instruct-2407
library_name: peft
tags:
- peft
- lora
- mistral
- mistral-nemo
- causal-lm
- checkpoint
---

# Nemo-Recwnt — Checkpoint 200

This repository contains an intermediate **LoRA/PEFT adapter checkpoint** for `mistralai/Mistral-Nemo-Instruct-2407`. It is not a fully merged standalone model; it is an adapter intended to be loaded on top of the base model. :contentReference[oaicite:0]{index=0}

## Model Summary

- **Base model:** `mistralai/Mistral-Nemo-Instruct-2407` :contentReference[oaicite:1]{index=1}
- **Adapter type:** PEFT / LoRA :contentReference[oaicite:2]{index=2}
- **Task type:** Causal language modeling (`CAUSAL_LM`) :contentReference[oaicite:3]{index=3}
- **Checkpoint step:** 200 (`global_step = 200`) :contentReference[oaicite:4]{index=4}
- **LoRA rank (`r`):** 16 :contentReference[oaicite:5]{index=5}
- **LoRA alpha:** 16 :contentReference[oaicite:6]{index=6}
- **LoRA dropout:** 0.05 :contentReference[oaicite:7]{index=7}

## What’s in this Folder

This checkpoint directory includes:

- `adapter_model.safetensors` — the LoRA adapter weights
- `adapter_config.json` — PEFT adapter configuration
- `tokenizer.json`, `tokenizer_config.json`, `special_tokens_map.json` — tokenizer assets
- `trainer_state.json` — training state metadata
- `training_args.bin` — saved training arguments
- `scheduler.pt`, `rng_state_*.pth` — optimizer/runtime state files
- `zero_to_fp32.py` — utility script often produced in distributed training exports :contentReference[oaicite:8]{index=8}

## LoRA Configuration

The adapter targets the following projection layers:

- `q_proj`
- `k_proj`
- `v_proj`
- `o_proj`
- `up_proj`
- `down_proj`
- `gate_proj` :contentReference[oaicite:9]{index=9}

This suggests the model was fine-tuned broadly across both attention and MLP projection modules rather than a minimal attention-only LoRA setup. :contentReference[oaicite:10]{index=10}

## Training Status

This appears to be an **intermediate training checkpoint**, not a final fully documented release:

- the folder is named `checkpoint-200`
- `trainer_state.json` shows `global_step: 200`
- `max_steps` is `5716`
- `best_model_checkpoint` is `null` :contentReference[oaicite:11]{index=11}

So this checkpoint looks like an early saved snapshot from a longer run rather than the final selected model. :contentReference[oaicite:12]{index=12}

## Usage

### Load with Transformers + PEFT

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_id = "mistralai/Mistral-Nemo-Instruct-2407"
adapter_id = "zaddyzaddy/Nemo-Recwnt/checkpoint-200"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype="auto",
    device_map="auto"
)

model = PeftModel.from_pretrained(base_model, adapter_id)

prompt = "Explain what this model is and how it should be loaded."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=200)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))