ZQN-12B (ZAQERIS/ZQN-12B)

48-Layer Bilingual Text Model Compressed with ZAQERIS M3-GPTQ for 8 GB GPUs

Official Model Name: ZQN-12B | Repository: ZAQERIS/ZQN-12B | Model Family: ZQN (ZaqionDialecticForCausalLM)
Developed by: ZAQERIS AI (Independent AI Research Group, Cairo, Egypt)
Predecessor: ZAQERIS-AXIOM (79.43B total / 12.39B active Sparse MoE, 36 layers, M3 3.16 bpp)
Training Origin: Trained from scratch by ZAQERIS AI; all pre-training and SFT datasets are kept strictly private in-house and are not distributed in this repository.
Tokenizer Ownership (External): This tokenizer is NOT ours. ZQN-12B reuses an external pre-existing 152,064-token multilingual BPE tokenizer (vocab_size = 152,064, <|User|>, <|Assistant|>, <think>, </think>, <|end▁of▁sentence|>) for compatibility only.
License: Apache 2.0
Total Physical Parameters: 15,546,031,104 (15.55B unfolded: 13.21B 48-layer backbone + 1.56B embed/head (14.77B dense core) + 0.75B DVMU + 21.02M LTBDD)
Quantization: ZAQERIS M3-GPTQ (3x8b 8D RVQ + Block-Hadamard $g=128$ + OBQ salience 0.5% (0.4% lm_head) + Block-8 Cholesky + row normalization, 3.2484 bpw)
Compressed Disk Footprint: 6.479 GiB (6,957,289,320 bytes across 5 Safetensors shards)
Verified Consumer 8 GB Runtime: 6.032 GiB loaded CUDA VRAM / 6.614 GiB peak eval VRAM / 25.47 tok/s (9-prompt Stage-4 live suite, prefill=1.373 s) on RTX 5060 Laptop GPU, 8 GB
Context Window: 131,072 tokens Native (128K) / 262,144 tokens Extended via YaRN (256K, $m_{\text{scale}} = 1.069315$)

License: Apache 2.0 Model Parameters Context Compression Research Group

Quickstart Notebook: Quickstart & Verification Notebook
Quantization Paper: ZAQERIS M3-GPTQ Technical Paper


نبذة عن النموذج والشفافية التقنية (Overview in Arabic)

ZQN-12B هو النموذج الثاني من مجموعة ZAQERIS AI البحثية المستقلة (القاهرة، مصر) بعد الإصدار الأول ZAQERIS-AXIOM:

  • التدريب والبيانات: تم تدريب نموذج ZQN-12B بالكامل من الصفر (Trained from Scratch) بواسطة ZAQERIS AI، مع الاحتفاظ بجميع بيانات التدريب والضبط الدقيق (Pre-training & SFT Corpora) خاصةً بنا داخلياً دون نشر أي ملفات تدريب في هذا المستودع العام.
  • التوكنايزر (ملكية خارجية): التوكنايزر المرفق ليس من تطويرنا ولا نملكه؛ بل يعتمد النموذج فقط على توكنايزر BPE خارجي جاهز (vocab_size = 152,064) ورموزه الخاصة القياسية (<|User|>, <|Assistant|>, <think>, </think>, <|end▁of▁sentence|>) لضمان التوافق البرمجي فقط.
  • الهيكلة والحجم الفعلي: رغم تسمية السلسلة التجارية ZQN-12B، تضم النسخة الحالية 48 طبقة (d_model = 5120, intermediate_size = 13824, 40 رأس استعلام / 8 رؤوس KV، head_dim = 128) بإجمالي 15,546,031,104 معامل (14.77B نواة كثيفة + 0.75B DVMU + 21.02M LTBDD = 15.55B)، مضغوطة عبر ZAQERIS M3-GPTQ (3.2484 bpw) إلى 6.479 GiB على القرص (6.032 GiB VRAM بعد التحميل).

1. Actual Capabilities & Known Limitations (ما يفعله النموذج فعلياً وحدوده)

Designed Capabilities (Text-Only Causal LM):

  • Bilingual Arabic & English Reasoning + Dialogue: Supports step-by-step <think>...</think> reasoning formatting and direct answers in Arabic and English.
  • Local Consumer 8 GB Deployment: Compressed with ZAQERIS M3-GPTQ (3.2484 bpw linear projections + CPU-offloaded INT8 embed_tokens on <= 9 GiB CUDA GPUs + INT4 G64 DVMU adapters), achieving 6.614 GiB peak VRAM and 25.47 tok/s (Stage-4 9-prompt suite).
  • Structured Compact Tool Markup (<call .../> / <plan ...>): Can emit structured XML-style tool calls that an external host or local ZaqionAgentToolbox can parse. Standalone ZaqionAgentToolbox implements opt-in local sh, linux_diag, read_file, write_file, and search only; browser, email/outbox, web search, and media tools require an external host orchestrator.
  • Deterministic Lightweight Helpers Included in Modeling Code:
    • ZaqionSelfCorrectionEngine: a lightweight rule/lookup pre-normalizer (COMMON_CORRECTIONS table + trailing خ->ك rule + mash-string rejection triggers), tested 10/10 on its specific 10-case unit file.
    • ZaqionDialecticDeliberationEngine (LTBDD): a 21.02M-parameter residual bottleneck block prior to lm_head.

Current Limitations (Realistic Scope):

  • Early Checkpoint / Modest Academic Benchmark Scores: Stage-3 40-layer reference benchmarks achieved 15.24% on HumanEval, 19.00% on GSM8K (100 tasks), 45.50% on MMLU (200 tasks), 55.50% on ARC-Challenge (200 tasks), and 58.33% (7/12) on tool selection. It is not a frontier SOTA model.
  • Stage-4 Quantitative Coverage: Stage-4 (48-layer) weights have been verified for compression (3.2484 bpw, mean_cos = 0.9731, calib_ppl = 10.6807), held-out perplexity (35.5500 vs 43.6063 in Stage-3), throughput (25.47 tok/s), and 2/5 tool-tag emission on live_eval_report.json agent prompts; the full 5-suite benchmark sweep (HumanEval/GSM8K/MMLU/ARC/Agent) has not yet been re-run on Stage-4.
  • No Built-In Web Search, Vision, or Audio Generation: Standalone ZQN-12B weights are text-only causal weights (search_enabled: false).
  • No Hardware Power Metering Recorded: config.json sets "power_metering_recorded": false; no fabricated J/token or USD/1M cost metrics are claimed.

2. Architecture & Parameter Accounting (verify_parameter_count_and_m4_expansion.py)

Component Exact Parameters Precision / Packing
48-Layer Dense Backbone (layers.0..47 attn + MLP) 13,212,893,184 (~13.21B) M3-GPTQ 3x8b (0.5% OBQ, normalize_rows)
Embeddings (embed_tokens) + Output Head (lm_head) 1,557,140,480 (~1.56B) INT8 per-row (embed_tokens) + M3-GPTQ 3x8b (lm_head, 0.4% OBQ)
DVMU Virtual Manifold Experts (layers.0..47) 754,974,720 (~0.75B) Group-64 Symmetric INT4
LTBDD Tri-Branch Deliberation Engine 21,022,720 (~21.02M) float16
Total Physical Unfolded Parameters 15,546,031,104 (15.55B) Total Disk Size: 6.479 GiB (5 Safetensors shards)

Shard Layout (6.479 GiB / 6,957,289,320 Bytes Total)

  • model-00001-of-00005.safetensors: 1,618,173,454 bytes (1.51 GiB)
  • model-00002-of-00005.safetensors: 1,616,174,450 bytes (1.51 GiB)
  • model-00003-of-00005.safetensors: 1,611,934,110 bytes (1.50 GiB)
  • model-00004-of-00005.safetensors: 1,628,396,864 bytes (1.52 GiB)
  • model-00005-of-00005.safetensors: 482,610,442 bytes (0.45 GiB)

3. Empirical Measurements & Offline Benchmarks (Verified Locally — Zero Synthetic Numbers)

Evaluated locally on NVIDIA GeForce RTX 5060 Laptop GPU (8 GB VRAM), Windows 11, Python 3.12.10, PyTorch 2.11.0+cu128:

Metric Observed Result Notes & Provenance
M3-GPTQ Effective Bitrate (Stage-4) 3.2484 bpw m3_compression_report.json (0.5% OBQ, damp=0.02, 16 RVQ iters)
Mean Matrix Weight Cosine (Stage-4) 0.9731 Across 48 layers + lm_head (32,768 calib tokens; calib_ppl=10.6807)
Loaded CUDA VRAM / Peak VRAM (Stage-4) 6.032 GiB / 6.614 GiB embed_tokens auto-hosted on CPU when GPU <= 9 GiB
Held-Out Perplexity (M3-GPTQ Stage-4) 35.5500 Improved from 43.6063 (Stage-3) on 148-token held-out sequence (prefill=1.373 s)
Decode Speed (Stage-4 Live Suite) 25.47 tok/s (9 prompts) NVRTC Hadamard-LUT + Static T=1 CUDA Graph
HumanEval (164/164 Stage-3 reference) 15.24% (25/164) Greedy pass@1 in isolated 5s subprocess (benchmark_report.json; Stage-4 sweep pending)
GSM8K (100/1,319 Stage-3 reference) 19.00% (19/100) Exact numeric match (benchmark_report.json; Stage-4 sweep pending)
MMLU (200/14,042 Stage-3 reference) 45.50% (91/200) Deterministic 1-pass A/B/C/D log-prob (benchmark_report.json; Stage-4 sweep pending)
AI2 ARC-Challenge (200/1,172 Stage-3 reference) 55.50% (111/200) Deterministic 1-pass log-prob (benchmark_report.json; Stage-4 sweep pending)
Arabic/English Agent Tool Suite (12 tasks, Stage-3) 58.33% (7/12) / 100% JSON Unseen multi-tool calling (benchmark_report.json; Stage-4 live emitted calls on 2/5 agent prompts)
ASSC Rule Normalizer Unit Suite 10 / 10 (100%) evaluation_evidence/self_correction_typo_benchmark.jsonl
RoPE KV-Cache Position Consistency 0.0e+00 drift scripts/test_rope_kv_cache_consistency.py
YaRN 256K Frequency Stability Verified (mscale = 1.069315) scripts/test_yarn_long_context.py
DVMU Active Tensor Divergence Verified scripts/test_dvmu_real_forward_pass.py

(Note: generation_config.json sets repetition_penalty = 1.05, do_sample = False, and eos_token_id = [151645, 151643]. Full ablation details are in EVALUATION_BENCHMARK_COMPARISON.md.)


4. Quickstart Usage (8 GB+ GPU or CPU)

import sys
import torch
from transformers import AutoTokenizer
from huggingface_hub import snapshot_download

local_dir = snapshot_download("ZAQERIS/ZQN-12B")
sys.path.insert(0, local_dir)

from modeling_zaqeris_zaqion_12b import ZaqionDialecticForCausalLM

# External 152,064-vocab BPE tokenizer reused for compatibility (not trained by ZAQERIS AI)
tokenizer = AutoTokenizer.from_pretrained(local_dir, trust_remote_code=True)
model = ZaqionDialecticForCausalLM.from_m3_pretrained(
    local_dir,
    device="cuda" if torch.cuda.is_available() else "cpu",
).eval()

# Optional deterministic pre-normalizer
noisy_prompt = "ما قدراتخ وما هو اسم النموذج؟"
correction = model.correct_prompt(noisy_prompt)
print("Normalized Prompt:", correction["corrected_prompt"])

# Generate response using ZQN-12B chat template (preventing duplicate BOS)
messages = [{"role": "user", "content": correction["corrected_prompt"]}]
formatted = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
inputs = tokenizer(formatted, return_tensors="pt", add_special_tokens=False).to(model.device)
with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=192,
        do_sample=False,
        repetition_penalty=1.05,
        eos_token_id=[151645, 151643],
    )
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

5. Technical Documentation & Verification Suite

Executable Verification Suite:

python scripts/verify_parameter_count_and_m4_expansion.py
python scripts/test_rope_kv_cache_consistency.py
python scripts/test_yarn_long_context.py
python scripts/test_dvmu_real_forward_pass.py
python evaluation_evidence/reproduce_m3_benchmarks.py --model Qwen/Qwen2.5-Coder-1.5B
Downloads last month
520
Safetensors
Model size
7B params
Tensor type
I32
·
F16
·
U16
·
I8
·
U8
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support