README.md

---
license: apache-2.0
base_model: Qwen/Qwen3-1.7B
tags:
- unsloth
- lora
- qwen3
- reasoning
- distillation
- chain-of-thought
datasets:
- mitroitskii/OpenR1-Math-220k-formatted
- Jackrong/Claude-opus-4.6-TraceInversion-9000x
- Jackrong/Kimi-K2.5-Reasoning-1M-Cleaned
- WithinUsAI/MiniMax_M2.7_Distilled_5k
- FreedomIntelligence/medical-o1-reasoning-SFT
- Modotte/CodeX-2M-Thinking
- trjxter/DeepSeek-V4-Pro-Reasoning-8000x
- nvidia/OpenCodeReasoning
- openai/gsm8k
language:
- en
pipeline_tag: text-generation
library_name: transformers
---

![Atem Logo](https://huggingface.co/EphAsad/Atem-1.7B/resolve/main/Logo.png)

# Atem-1.7B

*Ancient logic. Modern intelligence.*

A 1.7B reasoning model trained via a single CoT-preserving SFT pass directly on Qwen3-1.7B, distilling multi-domain reasoning capability from frontier teacher models while keeping the base model's native thinking capability intact.

![Base Model](https://img.shields.io/badge/Base-Qwen3--1.7B-blue)![Method](https://img.shields.io/badge/Method-CoT--Preserving%20SFT-purple)![Parameters](https://img.shields.io/badge/Parameters-1.7B-orange)![License](https://img.shields.io/badge/License-Apache%202.0-green)

---

## Overview

Atem-1.7B is a 1.7B parameter reasoning model built via a single supervised fine-tuning pass on raw Qwen3-1.7B, using the same CoT-preserving single-pass design as Atem-4B and Atem-8B. It is the most compute-efficient model in the Atem series, completing training in under 2.5 hours on an A100-SXM4 80GB while maintaining 2.95% proportional LoRA capacity — close to the series-wide 3% target.

This model includes GSM8K-format training examples (5K no-think records) to partially restore the `####` answer convention that the reasoning corpus otherwise overwrites — an improvement over Atem-4B and Atem-8B, which did not include these.

---

## Model Details

| Property | Value |
| --- | --- |
| **Base model** | Qwen/Qwen3-1.7B |
| **Training method** | Single-pass CoT-Preserving LoRA SFT |
| **LoRA config** | r=48, alpha=96, dropout=0.05 |
| **Target modules** | q, k, v, o, gate, up, down projections |
| **Parameters** | ~1.77B |
| **Trainable (LoRA) params** | 52,297,728 (2.95% of base) |
| **Training records** | 62,301 (after token-length filtering) |
| **Think / No-think split** | 85% / 15% |
| **Epochs** | 2 (ceiling; early stopping patience=3, never triggered) |
| **Effective batch size** | 64 (batch 16 × grad accum 4) |
| **Learning rate** | 1e-4, cosine schedule, 5% warmup |
| **Max sequence length** | 6,144 tokens |
| **Precision** | bfloat16 (full 16-bit LoRA, not QLoRA) |
| **Hardware** | NVIDIA A100-SXM4 80GB |
| **Runtime** | 2h28m |
| **License** | Apache 2.0 |

---

## Design Notes

**Single combined pass.** The same single CoT-preserving pass design used across Atem-4B and Atem-8B — no erase-then-rebuild pipeline. Reasoning capability is built directly on the base model's intact native foundation.

**r=48 for proportional capacity.** r=32 on a 1.7B model represents only 2.05% of the model's parameters — the same shrinking-fraction problem observed across the series as model size grows. r=48 recovers 2.95% proportional capacity, close to the series-wide ~3% target and significantly better than r=32 would have provided.

**GSM8K format restoration.** The standard Atem training corpus uses `\boxed{}` notation throughout. Atem-4B and Atem-8B both showed a systematic GSM8K strict-match regression as a result of this format shift. Atem-1.7B is the first in the series to include 5,000 GSM8K-format training examples (from `openai/gsm8k`) in the no-think pool, partially re-establishing the `#### answer` convention alongside `\boxed{}`.

**Full 16-bit LoRA.** At 1.7B the model weights occupy only ~3.4GB, leaving over 75GB of A100 headroom. Full 16-bit LoRA is used throughout — faster and marginally more accurate than QLoRA without any VRAM constraint.

---

## Intended Use

Atem-1.7B is suited for reasoning tasks on resource-constrained hardware — edge devices, local deployment, and applications where a 4B+ model is impractical:

- Multi-step mathematical reasoning
- Code explanation, implementation, and debugging
- Analytical reasoning across diverse domains
- Commonsense reasoning and physical intuition
- Logic and argument evaluation

For higher capability at the cost of resource requirements, Atem-4B and Atem-8B provide progressively stronger results on the same reasoning tasks.

---

## Training Data

Atem-1.7B was trained on the same eight-source reasoning corpus as Atem-4B and Atem-8B, with the addition of 5,000 GSM8K-format records to partially restore the `####` answer convention. All sources include explicit chain-of-thought reasoning traces; 85% of training records were formatted with full think traces and 15% as direct answers.

| Dataset | Records | Source / Teacher |
| --- | --- | --- |
| mitroitskii/OpenR1-Math-220k-formatted | ~10,938 | DeepSeek-R1 — Mathematics (correctness-filtered) |
| Jackrong/Claude-opus-4.6-TraceInversion-9000x | 7,000 | Claude Opus 4.6 — Trace Inversion |
| Jackrong/Kimi-K2.5-Reasoning-1M-Cleaned (General-Math) | 8,000 | Kimi K2.5 — Mathematical Reasoning |
| Jackrong/Kimi-K2.5-Reasoning-1M-Cleaned (General-Distillation) | 8,000 | Kimi K2.5 — General Reasoning |
| Jackrong/Kimi-K2.5-Reasoning-1M-Cleaned (PHD-Science) | 8,000 | Kimi K2.5 — Scientific Reasoning |
| WithinUsAI/MiniMax_M2.7_Distilled_5k | 5,000 | MiniMax M2.7 |
| FreedomIntelligence/medical-o1-reasoning-SFT | 7,500 | Medical reasoning (English config) |
| Modotte/CodeX-2M-Thinking | 15,000 | Mixed — Coding with CoT |
| trjxter/DeepSeek-V4-Pro-Reasoning-8000x | ~8,014 | DeepSeek-V4-Pro |
| nvidia/OpenCodeReasoning | 15,000 | Mixed — Competitive coding |
| openai/gsm8k (no-think) | 5,000 | GSM8K `#### answer` format restoration |
| **Total (pre-filter pool)** | **96,017** | |
| **Total (post-filter, trained on)** | **62,301** | |

Non-English reasoning traces (primarily CJK) were filtered at the trace level using an ASCII-ratio threshold and retained as no-think records. The 34.3% filter rate is consistent with Atem-4B (32.7%) and Atem-8B (34.3%) at the same 6,144-token ceiling.

---

## Training Configuration

```python
# Key hyperparameters
lora_r             = 48
lora_alpha         = 96
lora_dropout       = 0.05
max_seq_length     = 6144
learning_rate      = 1e-4
lr_scheduler       = 'cosine'
warmup_ratio       = 0.05
batch_size         = 16
grad_accumulation  = 4           # effective batch size: 64
num_epochs         = 2           # ceiling — early stopping patience=3
eval_steps         = 150
early_stopping_patience   = 3
early_stopping_threshold  = 0.001
nothink_ratio      = 0.15
load_in_4bit       = False       # full 16-bit LoRA
dtype              = bfloat16
```

---

## Loss Curve

| Step | Train Loss | Val Loss |
| --- | --- | --- |
| 150 | 1.0706 | 1.0833 |
| 300 | 1.0385 | 1.0520 |
| 450 | 1.0566 | 1.0372 |
| 600 | 0.9990 | 1.0255 |
| 750 | 1.0082 | 1.0158 |
| 900 | 0.9887 | 1.0091 |
| 1050 | 0.9294 | 1.0051 |
| 1200 | 0.8906 | 1.0020 |
| 1350 | 0.9331 | 0.9993 |
| 1500 | 0.9780 | 0.9973 |
| 1650 | 0.9467 | 0.9963 |
| 1800 | 0.9341 | 0.9957 |
| Final (1948) | **0.9902** (avg) | **0.9956** |

Train loss is noisier than in larger Atem models — characteristic of smaller models with a diverse multi-domain corpus. Validation loss improved monotonically across all 13 checkpoints without exception. Early stopping was configured but never triggered.

---

## Evaluation

### Benchmark Results

Evaluated against base Qwen3-1.7B (`Qwen/Qwen3-1.7B`) using lm-evaluation-harness. Both models were loaded in 4-bit for evaluation. Statistical significance (σ) is provided as context for interpreting each result — at 1.7B scale, several deltas that appear directionally positive are within sampling noise due to test set size.

| Task | Base (Qwen3-1.7B) | Atem-1.7B | Delta | σ |
| --- | --- | --- | --- | --- |
| ARC-Challenge (0-shot, acc_norm) | 40.7% | 42.2% | +1.5pp ✓ | 0.7σ |
| GSM8K strict (5-shot, exact_match) | 62.0% | 58.7% | −3.3pp ⚠ | 1.7σ |
| HellaSwag (0-shot, acc_norm) | 59.4% | **61.3%** | **+1.9pp** ✓ | 2.8σ |
| MMLU (0-shot, acc) | 55.4% | 56.2% | +0.8pp ✓ | 1.3σ |
| Winogrande (0-shot, acc) | 61.8% | 61.1% | −0.7pp ⚠ | 0.4σ |
| PIQA (0-shot, acc) | 71.4% | 71.4% | +0.0pp — | 0.0σ |
| OpenBookQA (0-shot, acc_norm) | 36.0% | **39.0%** | +3.0pp ✓ | 1.0σ |
| BoolQ (0-shot, acc) | 76.5% | 76.0% | −0.5pp — | 0.5σ |

**HellaSwag (+1.9pp, 2.8σ)** is the only clearly statistically significant positive result. It uses normalised log-likelihood scoring over multiple-choice options — format-independent and not influenced by generation style. This is also the most consistent signal across the full Atem series (1.7B: +1.9pp, 4B: +2.9pp, 8B: +1.7pp), confirming genuine commonsense reasoning transfer from the CoT training corpus.

**OpenBookQA (+3.0pp)** is directionally strong but the test set is only 500 questions, giving 1.0σ — treat this as encouraging rather than conclusive.

**Winogrande (−0.7pp, ⚠)** despite the flag is 0.4σ and statistically indistinguishable from noise. Not a meaningful regression.

**MMLU (+0.8pp, 1.3σ)** is borderline. Consistent with the series pattern — neither model has a knowledge breadth advantage after CoT training.

Results at 1.7B are generally less pronounced than at 4B and 8B, as expected: smaller models with proportionally larger parameter changes per training step exhibit noisier benchmark behaviour, and the absolute capability headroom above random baselines is narrower.

### GSM8K — Formatting Shift

The strict-match regression (−3.3pp) follows the same pattern established at 4B and 8B: the training corpus uses `\boxed{}` notation, systematically shifting away from the `####` format that lm_eval's strict-match extraction expects. At 1.7B the base model scores 62.0% — above the threshold where formatting effects dominate over raw capability gains (the 0.6B base at 26.7% was below this threshold and actually improved on strict-match).

Atem-1.7B is the first model in the series to include GSM8K-format (`#### answer`) training examples. At 5,000 records out of 62,301 total (8%), this partially offsets the shift but does not eliminate it — larger proportions would be needed for full recovery. Based on the flexible-extraction recovery rate confirmed at 8B (68% of regression recovered), the estimated true capability gap is approximately −1.1pp rather than −3.3pp.

---

## Usage

### Transformers

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "EphAsad/Atem-1.7B"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

messages = [
    {
        "role": "user",
        "content": "Explain why the harmonic mean is used for average speeds rather than the arithmetic mean."
    }
]

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)

with torch.no_grad():
    output = model.generate(
        input_ids=inputs,
        max_new_tokens=2000,
        temperature=0.6,
        top_p=0.95,
        top_k=20,
        do_sample=True,
        repetition_penalty=1.1,
    )

response = tokenizer.decode(
    output[0][inputs.shape[1]:],
    skip_special_tokens=True
)
print(response)
```

### Unsloth (faster inference)

```python
from unsloth import FastLanguageModel
import torch

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="EphAsad/Atem-1.7B",
    max_seq_length=6144,
    dtype=torch.bfloat16,
    load_in_4bit=True,
)
FastLanguageModel.for_inference(model)

messages = [
    {
        "role": "user",
        "content": "What is the time complexity of merge sort and why?"
    }
]

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt"
).to("cuda")

with torch.no_grad():
    output = model.generate(
        input_ids=inputs,
        max_new_tokens=2000,
        temperature=0.6,
        top_p=0.95,
        top_k=20,
        do_sample=True,
    )

print(tokenizer.decode(
    output[0][inputs.shape[1]:],
    skip_special_tokens=True
))
```

### Ollama

```bash
# Recommended — best speed/quality balance
ollama run hf.co/EphAsad/Atem-1.7B:Q4_K_M

# Higher quality
ollama run hf.co/EphAsad/Atem-1.7B:Q5_K_M

# Near-lossless
ollama run hf.co/EphAsad/Atem-1.7B:Q8_0
```

### llama.cpp

```bash
llama-server -hf EphAsad/Atem-1.7B:Q4_K_M
```

### Sampling Parameters

Use `temperature=0.6, top_p=0.95, top_k=20` — Qwen3's published recommendation for thinking mode. Do not use greedy decoding with thinking mode enabled.

### System Prompt

Atem-1.7B's identity is baked into the chat template and activates automatically without an explicit system message. For manual override:

```
You are Atem, a precise and analytical reasoning assistant. You approach
every problem methodically — identifying core concepts, reasoning step by
step, and arriving at well-supported conclusions. You show your thinking
clearly and are thorough, direct, and intellectually honest.
```

### Available Files

| File | Size | Description |
| --- | --- | --- |
| `model.safetensors` | 3.44 GB | Full bfloat16 merged weights (single shard) |
| `Atem-1.7b.Q4_K_M.gguf` | 1.11 GB | 4-bit quantised — recommended |
| `Atem-1.7b.Q5_K_M.gguf` | 1.26 GB | 5-bit quantised |
| `Atem-1.7b.Q8_0.gguf` | 1.83 GB | 8-bit quantised — near-lossless |

---

## Known Limitations

**GSM8K formatting shift.** As documented in the evaluation section, the training corpus uses `\boxed{}` for mathematical answers. Despite the inclusion of 5,000 GSM8K-format examples, the strict-match regression persists at −3.3pp. The estimated true capability gap under flexible extraction is approximately −1.1pp. Future runs with a higher proportion of GSM8K-format examples would reduce this further.

**Statistical modesty at 1.7B.** Most benchmark deltas at this scale are within sampling noise — HellaSwag is the exception (2.8σ). This is expected: 1.7B models have narrower performance headroom and proportionally larger variance per benchmark question. The reasoning improvements are real but harder to detect reliably at smaller scale.

**6,144 token sequence ceiling.** The longest reasoning traces (advanced mathematics, competitive programming) were dropped during formatting. The model has not been trained on very long chain-of-thought traces.

**No RLHF or DPO.** Atem-1.7B has not undergone preference optimisation.

---

## Roadmap

- **Atem-14B:** Single CoT-preserving pass on Qwen3-14B, r=128 (3.10% proportional capacity), with expanded GSM8K-format and camel-ai/chemistry additions to the corpus

---

## Citation

```bibtex
@misc{atem_1b7_2026,
  author       = {Asad, Zain},
  title        = {Atem-1.7B: A 1.7B CoT-Preserving Reasoning Model via
                  Single-Pass SFT on Qwen3},
  year         = {2026},
  publisher    = {HuggingFace},
  howpublished = {\url{https://huggingface.co/EphAsad/Atem-1.7B}},
}
```

---

## License

Released under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0), consistent with the base model Qwen/Qwen3-1.7B.

---

Built independently by Zain Asad — [EphAsad](https://huggingface.co/EphAsad)