test-v3-sft-grpo

This model is a fine-tuned version of amu870/test-v3 using GRPO (Group Relative Policy Optimization) via the TRL library.

Training Pipeline

SFT → DPO → GRPO (this model)

Training Objective

This model has been optimized using GRPO with a parse-based reward function that validates structured outputs (JSON, YAML, TOML, CSV, XML).

Training Configuration

• Base model (DPO merged): amu870/test-v3
• Method: GRPO (Group Relative Policy Optimization)
• Epochs: 1
• Learning rate: 1e-07
• Beta (KL penalty): 0.05
• Generations per prompt: 8
• LoRA Config: r=8, alpha=16

Reward Function

The reward function validates whether the generated output can be successfully parsed as the target format:

• Parse success: 1.0
• Parse failure: 0.0

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "amu870/test-v3-sft-grpo"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto"
)

# Test inference
prompt = "Extract the following attributes from text and output JSON..."
inputs = tokenizer.apply_chat_template(
    [{"role": "user", "content": prompt}],
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt"
).to("cuda")
outputs = model.generate(inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0]))

License

Apache 2.0. Users must follow the original base model's license terms.