README.md

---
license: mit
datasets:
- ai-factory/red_pajama_subset_arxiv_subset
- ai-factory/glaiveai-reasoning-v1-20m-chat
base_model:
- meta-llama/Llama-3.2-3B
library_name: adapter-transformers
---
# 🚀 Full Finetuned LLaMA 3.2 3B for AI Factory

This model combines the base `full_finetuned_llama3b` with LoRA fine-tuning on:
- `ai-factory/red_pajama_subset_arxiv_subset`
- `ai-factory/glaiveai-reasoning-v1-20m-chat`

- ✅ Tokenizer: ai-factory/giant
- 🔗 Adapter format: QLoRA (PEFT)
- 🧪 Torch dtype

## 🔍 Usage
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("your-hf-username/full_finetuned_llama3b")
tokenizer = AutoTokenizer.from_pretrained("ai-factory/giant")
# Load base model
base_model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL,
    torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32,
    device_map="auto",
    trust_remote_code=True,
    use_safetensors=True,
    local_files_only=True
)

# Apply LoRA
peft_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=8,
    lora_alpha=32,
    lora_dropout=0.05,
    bias="none",
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"]
)
model = get_peft_model(base_model, peft_config)
model.eval()
if torch.cuda.is_available():
    model = model.cuda()

# Load streaming datasets
arxiv = load_dataset("ai-factory/red_pajama_subset_arxiv_subset", split="train", streaming=True)
glaive = load_dataset("ai-factory/glaiveai-reasoning-v1-20m-chat", split="train", streaming=True)

def tokenize(example):
    return tokenizer(example["text"], truncation=True, max_length=4096)

# Tokenize small samples
tokenized_arxiv = map(tokenize, islice(arxiv, args.sample_size))
tokenized_glaive = map(tokenize, islice(glaive, args.sample_size))

# Run forward + backward pass (init LoRA weights)
print("🔥 Training one step to initialize LoRA...")
for i, sample in enumerate(tokenized_arxiv):
    if not sample.get("input_ids"):
        continue
    ids = torch.tensor(sample["input_ids"]).unsqueeze(0).to(model.device)
    labels = ids.clone()
    loss = model(input_ids=ids, labels=labels).loss
    loss.backward()
    break

# Merge LoRA and save
print("🔁 Merging adapter into base model...")
merged_model = model.merge_and_unload()
merged_model.save_pretrained(SAVE_DIR, safe_serialization=True)
tokenizer.save_pretrained(SAVE_DIR)
print(f"✅ Merged model saved to {SAVE_DIR}")
```

## 👤 Authors
- AI Factory Miner Submission

## 📚 License
- Meta LLaMA license