ariacompute/qwen3-1.7b_q8

Model Card for Qwen3-1.7B (Aria Quant Bundle, q8)

Model Details

Model Description

Qwen3 1.7B is a 1.7-billion-parameter, dense Transformer decoder-only language model developed by the Qwen team at Alibaba Cloud, pre-trained on diverse public corpora and aligned via supervised fine-tuning (SFT) and direct preference optimization (DPO). This distribution is provided by Aria Compute as an aria-quant-bundle — a uniform 8-bit quantized package using Hadamard rotation + Lloyd-Max codebook quantization with per-group codebooks (group size 32). It delivers near-lossless generation quality — on qwen3-0.6B, logprob delta is just +0.00685 above FP16 and exact prefix fraction ties the best quant recipe at 0.3854 — at 1.8× smaller than FP16 (3.4 GB → ~1.9 GB). Optimized for CPU-only, on-device inference on mobile phones, edge devices, and single-board computers via the Aria Engine runtime. No GPU or cloud connection is required.

• Developed by: Qwen Team (Alibaba Cloud)
• Quantized and distributed by: Aria Compute
• Model type: Dense Transformer decoder-only (language)
• Language(s): English (primary), Chinese, and 20+ additional languages
• License: Apache 2.0
• Finetuned from model: Qwen/Qwen3-1.7B

Model Sources

• Original Repository: QwenLM/Qwen3
• Original Paper: Qwen3 Technical Report (pending)
• Aria Compute Dashboard: ariacompute.com/dashboard/models
• Aria Engine: ariacompute.com

Uses

Direct Use

This quantized bundle is intended for on-device, offline text-generation tasks on resource-constrained hardware, including:

• On-device chat and conversational assistants
• Real-time text completion and sentence prediction
• Structured tool calling / function calling for mobile and IoT APIs
• Lightweight text embeddings for on-device retrieval and classification
• Short-form summarization of notifications, messages, and local content

All inference runs locally on CPU. No data is sent to external servers.

Target Devices

Platform	Runtime Memory	Feasibility
High-end smartphone (8 GB)	~2.06 GB	✅ Recommended
Mid-range smartphone (4–6 GB)	~2.06 GB	✅
Budget phone (2–3 GB)	~2.06 GB	⚠️ Tight, but functional
Raspberry Pi 5 / SBC (4–8 GB)	~2.06 GB	✅
IoT gateway (1–2 GB)	~2.06 GB	❌ Insufficient
Wearable (1 GB)	~2.06 GB	❌ Insufficient

Memory breakdown (q8, at 4K context): ~1.9 GB quantized model weights (mmap) + ~112 MB KV cache + ~50 MB runtime overhead ≈ ~2.06 GB.

Note: The KV cache size is identical to qwen3-0.6b (28 layers × 8 KV heads × head_dim=128), as GQA configuration is shared across the Qwen3 family.

Out-of-Scope Use

• Long-form creative writing (>2K tokens per generation)
• Mathematical theorem proving or complex multi-step reasoning
• Full program synthesis (reliable for short functions only)
• Multimodal input (this model is text-only)
• Real-time audio/speech processing (use Aria speech models)
• Safety-critical decision systems without human oversight
• Deployment in production when batch inference or GPU acceleration is required (this bundle targets CPU-only, single-prompt inference)

How to Get Started with the Model

Download from Aria Compute

Authenticated dashboard users can download the bundle via: https://ariacompute.com/dashboard/models

Quantization Recipe

This bundle uses the uniform 8-bit quantization recipe — the maximum-fidelity / near-lossless option in the Aria Compute lineup:

Component	Quantization Strategy	Details
Attention Q/K/V/O weights	8-bit	Per-group codebooks (group_size=32), Hadamard pre-processing
FFN up/gate/down weights	8-bit	Per-group codebooks (group_size=32), Hadamard pre-processing
RMSNorm weights	FP16	Preserved at full precision
Embedding table	FP16	Preserved at full precision (tied input/output vocab)

• Bundle size: ~1.9 GB (FP16 original: ~3.4 GB, ~1.8× compression)
• Generation quality: Awaiting gen_quant_eval audit. Method reference (qwen3-0.6b_q8+group): token overlap 0.6429, exact prefix fraction 0.3854 (ties best), logprob delta +0.00685 — near-lossless, statistically indistinguishable from FP16
• Calibration-free: Hadamard rotation + Lloyd-Max codebook, no task-specific calibration data required
• Maximum fidelity recipe: This is the highest-quality quantized option for Qwen3-1.7B. For a smaller bundle with comparable prefix quality, consider qwen3-1.7b_q326_channel (recommended generation-quality recipe). For the smallest practical bundle, use qwen3-1.7b_q4

Model Architecture

Qwen3-1.7B employs a standard dense Transformer decoder architecture:

Parameter	Qwen3-0.6B	Qwen3-1.7B
Layers	28	28
Hidden size	1,024	2,048
FFN intermediate size	2,816	6,144
Attention heads (Query)	16	16
Attention heads (KV)	8 (GQA, group 2)	8 (GQA, group 2)
Head dimension	128	128
Activation	SiLU (SwiGLU-style gating)	SiLU (SwiGLU-style gating)
Position encoding	RoPE (θ = 1,000,000)	RoPE (θ = 1,000,000)
Normalization	RMSNorm (pre-norm)	RMSNorm (pre-norm)
Vocabulary size	151,936	151,936
Max context length	32,768	40,960
Input/output vocab	Tied	Tied

Design highlights (shared with Qwen3 family):

• GQA (Grouped Query Attention): 8 KV heads serving 16 query heads — halves KV Cache memory
• RoPE high base frequency (1M): Native support for up to 40K context length (32K to 40K depending on member)
• Dense FFN + SiLU gating: High inference efficiency, suitable for on-device use
• Tied vocab: Input embedding and output projection weights are shared, saving ~311M parameters

Key architecture differences from Qwen3-0.6B:

• Double hidden width (2048 vs 1024) and ~2.2× FFN width (6144 vs 2816) — the primary sources of the 1.1B parameter increase while keeping the same 28-layer depth
• 40K max context vs 32K — allows longer document processing at higher memory cost for extreme sequence lengths

Bias, Risks, and Limitations

Limitations

• Reasoning depth: Multi-step logical reasoning (≥3 steps) benefits from the larger capacity compared to 0.6B-class models, but still falls short of 7B+ frontier models. Verify outputs in high-stakes scenarios.
• Mathematics: GSM8K and MATH performance is improved over 0.6B models, but remains modest. Use larger models for quantitative tasks requiring precision.
• Code generation: Capable of short function completions and multi-line snippets; unreliable for multi-file synthesis or algorithmic problem solving.
• Factual knowledge: Improved world knowledge over 0.6B models due to wider hidden dimension and larger FFN capacity, but still limited compared to larger models. Always verify factual claims against authoritative sources.
• Instruction following: Handles moderate multi-constraint prompts reliably at typical context lengths. Keep complex, highly constrained instructions within 2-3 constraints.
• Quantization drift: As the near-lossless 8-bit recipe, generation consistency is expected to be essentially indistinguishable from FP16. On qwen3-0.6b_q8, logprob delta is just +0.00685 and exact prefix fraction ties the best at 0.3854. Greedy sampling can still fork on ambiguous continuations.

Bias and Risks

• Bias: As with all large language models trained on web-scale data, Qwen3 may reflect societal biases present in its training corpus. Evaluate outputs before deployment in sensitive domains (hiring, healthcare, law).
• Toxicity: The base model has been safety-aligned with refusal training. However, no safety filter is exhaustive. Consider an additional output classifier in high-risk environments.
• Hallucination: May generate plausible-sounding but factually incorrect information. Implement output verification for critical applications.
• Dual-use risk: Text-generation capabilities could be misused for spam, disinformation, or impersonation. Deploy responsibly and in accordance with the Apache 2.0 license terms.

Recommendations

Users (both direct and downstream) should be made aware of the above risks, biases, limitations, and constraints of the model. We recommend:

• Adding a lightweight output safety classifier for user-facing deployments
• Verifying factual claims with external knowledge bases
• Not using the model for high-stakes decisions without human review