| --- |
| license: apache-2.0 |
| language: |
| - en |
| tags: |
| - text-generation |
| - causal-lm |
| - pytorch |
| - sft |
| - instruction-tuned |
| - chat |
| - reasoning |
| - tool-use |
| - hybrid |
| - gated-deltanet |
| - gqa |
| - tercet |
| pipeline_tag: text-generation |
| library_name: tiny_gdn |
| datasets: |
| - HuggingFaceFW/fineweb-edu |
| - HuggingFaceTB/smoltalk2 |
| - HuggingFaceTB/smoltalk |
| - nvidia/Llama-Nemotron-Post-Training-Dataset |
| - nvidia/Nemotron-Post-Training-Dataset-v1 |
| - nvidia/Nemotron-Agentic-v1 |
| - nvidia/Nemotron-SFT-Agentic-v2 |
| - nvidia/Nemotron-SFT-Instruction-Following-Chat-v2 |
| - nvidia/Nemotron-Instruction-Following-Chat-v1 |
| - nvidia/Nemotron-Cascade-2-SFT-Data |
| - Agent-Ark/Toucan-1.5M |
| - interstellarninja/hermes_reasoning_tool_use |
| - NousResearch/Hermes-3-Dataset |
| - allenai/WildChat-4.8M |
| - open-thoughts/OpenThoughts3-1.2M |
| - teknium/OpenHermes-2.5 |
| - open-r1/Mixture-of-Thoughts |
| - CohereLabs/aya_dataset |
| - NousResearch/hermes-function-calling-v1 |
| - simplescaling/s1K-1.1 |
| - allenai/tulu-3-sft-personas-instruction-following |
| - Salesforce/xlam-function-calling-60k |
| - THUDM/LongAlign-10k |
| base_model: kerzgrr/Tercet-R-1.0 |
| --- |
| |
| <div align="center"> |
|
|
| # Tercet-R-1.1 |
|
|
| ### Reasoning + tool-call chat model (~502M) — latest Tercet-R snapshot |
|
|
| [](.) |
| [](.) |
| [](LICENSE) |
| [](https://huggingface.co/spaces/kerzgrr/tercet-r-1.1-demo) |
|
|
| *A ~502M hybrid GDN-2 + GQA model, continued on multi-turn tools and instruction following* |
|
|
| </div> |
|
|
| --- |
|
|
| ## What this is |
|
|
| **Tercet-R-1.1** is the latest public reasoning / tool-use checkpoint in the Tercet-R line. |
|
|
| - Previous public snapshot: [`kerzgrr/Tercet-R-1.0`](https://huggingface.co/kerzgrr/Tercet-R-1.0) |
| - Base (pretrain): [`kerzgrr/Tercet-base`](https://huggingface.co/kerzgrr/Tercet-base) |
| - Live demo: [`kerzgrr/tercet-r-1.1-demo`](https://huggingface.co/spaces/kerzgrr/tercet-r-1.1-demo) |
| - Hub weights are the **EMA** snapshot in **bfloat16** |
|
|
| This upload is the latest **complete** stage-3 SFT checkpoint (optimizer step **5,300**). The stage-3 run is still in progress toward a much longer schedule; 1.1 is the snapshot that existed when this repo was published. |
|
|
| Initialization was the instruct-run EMA at step 5,000 (`tercet-r-instruct`), not the public 1.0 file (that release packaged instruct step 4,500). The chat contract is unchanged. |
|
|
| --- |
|
|
| ## Chat contract |
|
|
| ### Thinking |
|
|
| Each assistant turn is prefixed with a **zero-loss control token**: |
|
|
| | Mode | Prefix | Typical body | |
| |------|--------|----------------| |
| | think | `<\|think\|>\n` | `<think>…</think>` then the answer | |
| | no-think | `<\|no_think\|>\n` | answer only | |
|
|
| `inference.py` streams the `<think>` region live (dim yellow) and hides the control tokens. |
|
|
| ### Tool calls (SmolTalk JSON) |
|
|
| ``` |
| <tool_call> |
| {"name": "web-search", "arguments": {"query": "…"}} |
| </tool_call> |
| ``` |
|
|
| Stage-3 also saw `stateful_python_code_exec` leftovers from earlier Cascade-2 SFT. `inference.py` auto-runs: |
|
|
| | Built-in | Tool name | Observation | |
| |----------|-----------|-------------| |
| | `--tools web-search` | `web-search` | Tavily-shaped JSON (Tavily if `TAVILY_API_KEY` is set, else DuckDuckGo + Wikipedia) | |
| | `--tools python` | `stateful_python_code_exec` | Jupyter-style stdout / last value from a restricted interpreter | |
| | `--tools calculator` | `calculator` | Numeric result of a math expression | |
|
|
| ### Tool results |
|
|
| Each observation is a `tool` (or `user`) turn prefixed with: |
|
|
| ``` |
| <|tool_response|> |
| {observation} |
| ``` |
|
|
| --- |
|
|
| ## Install & run |
|
|
| ```bash |
| pip install torch safetensors tokenizers huggingface_hub |
| hf download kerzgrr/Tercet-R-1.1 inference.py --local-dir . |
| python inference.py |
| ``` |
|
|
| ```bash |
| python inference.py --prompt "What is the capital of France?" |
| python inference.py --tools web-search,python,calculator |
| python inference.py --no-think --prompt "Reply in one sentence." |
| ``` |
|
|
| `inference.py` auto-downloads weights / tokenizer / `tiny_gdn/` and auto-installs pinned `flash-linear-attention`. Git is required on `PATH`. |
|
|
| | Flag | Default | Description | |
| |------|---------|-------------| |
| | `--prompt` | — | One-shot user message | |
| | `--system` | — | System prompt, used verbatim | |
| | `--think` / `--no-think` | think | Assistant control prefix | |
| | `--tools` | — | Built-ins: `web-search`, `python`, `calculator` (comma-separated) | |
| | `--temperature` | `0.7` | Sampling temperature | |
| | `--max-new-tokens` | `4096` | Max generation length | |
| | `--device` | `cuda` if available | `cuda` / `cpu` | |
|
|
| Interactive commands: `/think` `/no_think` `/system …` `/reset` `/exit`. |
|
|
| --- |
|
|
| ## Model architecture |
|
|
| Same TinyGDN hybrid as [`Tercet-base`](https://huggingface.co/kerzgrr/Tercet-base) (501,635,264 parameters): |
|
|
| | | | |
| |--|--| |
| | **Layers** | 32 (GDN-2 ×3 + GQA every 4th) | |
| | **Hidden** | 1,024 | |
| | **MLP** | SwiGLU 2,624 | |
| | **Attention** | 8 Q / 2 KV, head dim 128, partial RoPE | |
| | **Linear** | Gated DeltaNet-2, 8 heads × 128 | |
| | **Vocab** | 49,152 BPE | |
| | **Context** | 16,384 | |
|
|
| --- |
|
|
| ## Training |
|
|
| | Stage | Details | |
| |-------|---------| |
| | **Base** | [HuggingFaceFW/fineweb-edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) → [`Tercet-base`](https://huggingface.co/kerzgrr/Tercet-base) | |
| | **Mid + instruct SFT** | [HuggingFaceTB/smoltalk2](https://huggingface.co/datasets/HuggingFaceTB/smoltalk2) Mid / SFT → instruct EMA @ step 5,000 | |
| | **Stage-3 SFT** | Multi-turn mix: Nemotron PTD v1, Agentic v1/v2 (including `web-search`), IF-Chat v1/v2, Cascade-2 instruction following, Toucan 1.5M, Hermes-3, Hermes reasoning tool-use, WildChat-4.8M. Seq 16,384, AdamW 3×10⁻⁵, 27.5 hours so far, 602M training tokens | |
| | **Checkpoint** | optimizer step 5,300 (latest complete stage-3 snapshot) | |
| | **Weights** | EMA (this repo's `model.safetensors`) | |
| | **Val loss (EMA)** | 2.027 (ppl 7.59) on the stage-3 holdout — a harder mix than the instruct-only 1.0 val set | |
|
|
| Stage-3 does **not** re-include Cascade-2 `math/math_tool.jsonl`. The restricted Python tool in `inference.py` is there so leftover `stateful_python_code_exec` calls still compute instead of failing open. |
|
|
| --- |
|
|
| ## Limitations |
|
|
| - **In-progress run**: this is a mid-stage-3 snapshot, not a finished 1.1 train |
| - **Scale**: ~502M is a research / edge model, not a frontier system |
| - Requires `flash-linear-attention`; not GGUF / llama.cpp compatible today |
| - The Python tool is a restricted interpreter (math-oriented imports only) |
|
|
| --- |
|
|
| ## Model family |
|
|
| | Model | Stage | Hub | |
| |-------|-------|-----| |
| | Tercet-base | Pretrain | [`kerzgrr/Tercet-base`](https://huggingface.co/kerzgrr/Tercet-base) | |
| | Tercet | SFT chat | [`kerzgrr/Tercet`](https://huggingface.co/kerzgrr/Tercet) | |
| | Tercet-R-1.0 | SFT reasoning + tools | [`kerzgrr/Tercet-R-1.0`](https://huggingface.co/kerzgrr/Tercet-R-1.0) | |
| | **Tercet-R-1.1** | **Stage-3 SFT (latest)** | **this repo** | |
| | Demo | ZeroGPU Space | [`kerzgrr/tercet-r-1.1-demo`](https://huggingface.co/spaces/kerzgrr/tercet-r-1.1-demo) | |
|
|
| --- |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{tercetr11, |
| title={Tercet-R-1.1: A 502M Hybrid GDN-2 + GQA Reasoning Model}, |
| author={kerzgrr}, |
| year={2026}, |
| url={https://huggingface.co/kerzgrr/Tercet-R-1.1} |
| } |
| ``` |
|
|
| --- |
|
|
| <div align="center"> |
|
|
| *R is for reasoning.* |
|
|
| </div> |
|
|
