ping98k/gpt-j-6b-thinking-sft-lora

GPT-J 6B — Thinking SFT · LoRA Adapter

Raw LoRA adapter (QLoRA, r=32, α=32) for EleutherAI/gpt-j-6b, trained with the Qwen3 chain-of-thought format.

The fully merged float16 model (ready to use without PEFT) is at
👉 ping98k/gpt-j-6b-thinking-sft

---

Adapter Details

Property	Value
Base model	EleutherAI/gpt-j-6b
LoRA rank	32
LoRA alpha	32
Target modules	q_proj, k_proj, v_proj, out_proj, fc_in, fc_out
Trainable parameters	~50 M
Vocabulary additions	`<

---

Usage with PEFT

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_id = "EleutherAI/gpt-j-6b"
adapter_id    = "ping98k/gpt-j-6b-thinking-sft-lora"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)  # resized vocab

model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.float16,
    device_map="auto",
)
model.resize_token_embeddings(len(tokenizer))
model = PeftModel.from_pretrained(model, adapter_id)
model.eval()

Tip: For most use cases, prefer the pre-merged model at ping98k/gpt-j-6b-thinking-sft — no PEFT dependency needed.

---

License

Apache 2.0 (inherits from EleutherAI/gpt-j-6b).