ANISH-j/Quantized-Llama-3.2-1B-15k

Orpo-Llama-3.2–1B-15k (Q4_K_M)

Quantized GGUF version of Orpo-Llama-3.2–1B-15k, optimized for efficient local inference using :contentReference[oaicite:0]{index=0}.

Model Overview

• Base Model: meta-llama/Llama-3.2–1B
• Fine-tuning: ORPO
• Training Data: 15k samples from mlabonne/orpo-dpo-mix-40k
• Parameters: ~1B
• Architecture: Llama (Transformer)

Quantization

• Method: Post-Training Quantization (PTQ)
• Format: GGUF
• Precision: Q4_K_M (4-bit K-quant)
• Tooling: llama.cpp
• Size Reduction: ~2.3 GB (FP16) → ~700–850 MB

Conversion Pipeline

1. Hugging Face .safetensors
2. GGUF F16 (intermediate)
3. GGUF Q4_K_M (final)

Technical Specs

• Layers: 16
• Context Length: 131072
• Embedding Size: 2048
• FFN Size: 8192
• Attention Heads: 32 (KV: 8)
• Vocab Size: 128,258
• RoPE Base: 500,000
• Quantization Version: 2

Usage

Designed for fast CPU / low-VRAM inference via llama.cpp-compatible runtimes.

Resources

• Colab (conversion reference):
  https://colab.research.google.com/drive/1bZOnFp01XcLaYYWt2xJ5LAzOD9msvEZL?usp=sharing