README.md

---
base_model: GSAI-ML/LLaDA-8B-Instruct
library_name: peft
pipeline_tag: text-generation
tags:
- peft
- lora
- llada
- math
---

# Math LoRA Adapter for LLaDA-8B-Instruct

This repository contains a PEFT LoRA adapter fine-tuned for math reasoning on top of
[`GSAI-ML/LLaDA-8B-Instruct`](https://huggingface.co/GSAI-ML/LLaDA-8B-Instruct).

The repository stores adapter weights only. It does not include the base model weights.

## Files

- `adapter_model.safetensors`: LoRA adapter weights.
- `adapter_config.json`: PEFT LoRA configuration.
- `tokenizer.json`, `tokenizer_config.json`, `special_tokens_map.json`: tokenizer files used with the adapter.
- `trainer_state.json`, `training_args.bin`: training metadata kept for traceability.

Optimizer, scheduler, and RNG checkpoint files are intentionally not included.

## Usage

```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model_id = "GSAI-ML/LLaDA-8B-Instruct"
adapter_id = "mousezhang/math-llada8b"

tokenizer = AutoTokenizer.from_pretrained(base_model_id, trust_remote_code=True)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    trust_remote_code=True,
)
model = PeftModel.from_pretrained(base_model, adapter_id)
```

## Adapter Details

- Base model: `GSAI-ML/LLaDA-8B-Instruct`
- Adapter type: LoRA via PEFT
- Task/domain: math reasoning
- PEFT task type: `CAUSAL_LM`
- LoRA rank: `128`
- LoRA alpha: `128`
- LoRA dropout: `0.05`
- Target modules: `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`

## Training and Evaluation

This model card does not report benchmark scores. Evaluation results should be treated as
not provided unless published separately by the model author.

## Limitations

This adapter inherits the limitations and license terms of the base model. It is intended
for research and experimental use, and outputs should be checked carefully before use in
high-stakes settings.