---
license: apache-2.0
base_model: Qwen/Qwen3.5-0.8B-Base
datasets:
- mohitkrconnect/heed-voice-notes-to-tasks
language:
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- on-device
- task-extraction
- structured-output
- json
- voice-notes
- small-language-model
- research-checkpoint
- negative-result
---

# Heed simulated notes + extra mistakes, Qwen3.5-0.8B (EXP-031, run E2): a negative result

> **Research checkpoint and negative result, not the app model.** The model inside Heed is [`heed-tasks-qwen3-0.6b`](https://huggingface.co/mohitkrconnect/heed-tasks-qwen3-0.6b). This one uses the **old, wide output format** (intent and voice commands as well as items) on the Qwen3.5-0.8B base. It was scored on the older guide-v2 yardstick, not by the blind judge, and it was never put on a phone.

## Why this model exists

At that stage the 0.8B student fine-tuned on Claude Sonnet 5.5's labels reached 47.2% of notes fully right on the yardstick (EXP-030, recipe "D3"). I could not supply more real notes. EXP-031 asked: does adding 1,109 simulated notes in my voice, and giving extra weight to the notes the model gets wrong where two teacher models agree, move the student closer to Sonnet, especially on real notes?

**E2 is the arm with both changes.** It tied the reference (47.2%) and no arm beat it. The answer to the question was no measurable gain (decision D101). The experiment also produced the study's seed-noise rule: the same D3 recipe scored 47.2% with seed 0 and 44.2% with seed 1, so single-run gaps below about 3 to 4 points are not results.

The related weight average of E2 and the seed-1 run is [`heed-simnotes-soup-qwen3.5-0.8b`](https://huggingface.co/mohitkrconnect/heed-simnotes-soup-qwen3.5-0.8b).

## Prompt and answer format (exact, old wide format)

Raw text, no chat template. The prompt lists the buckets, then the note, and ends with `JSON:`.

```
Categories:
- tasks (General): general things to do
- buy (Buy): things to purchase
- watch (Watch): movies, shows, videos to see
- ideas (Ideas): thoughts to remember
- work (Work): office and job tasks
- journal (Journal): feelings, emotions, stories with no action
Note: <the note text>
JSON:
```

The answer is JSON followed by EOS. An invented example:

```
{"i":<intent>,"items":[{"t":"Call the bank","c":"tasks","r":1,"w":"at 5"}],"cmd":[...]}
```

| Key | Meaning |
|---|---|
| `i` | An intent label for the whole note: `capture`, `mixed`, `empty`, `journal_only`, `command` or `question` |
| `items` | The tasks: `t` text, `c` bucket id, `r` reminder 0 or 1, `w` spoken time words (only when there is a reminder), optional `s`:1 for "suggested" |
| `cmd` | Voice commands: `a` = `complete`, `query`, `delete`, `move`, `reschedule` or `rename`, with `x` the target and `to` the destination |

There is no "Recorded:" date line. The default bucket list above is the exact training text (from the data builder); the prompt ends with a newline after `JSON:`. Intent and commands are parked in the app, and this wide format is not the app's format today.

### transformers

Qwen3.5 needs a transformers release with Qwen3.5 support (5.20.2 was used). Training used `flash-linear-attention` 0.3.2 and `causal-conv1d` 1.7.0 for speed.

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "mohitkrconnect/heed-simnotes-e2-qwen3.5-0.8b"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16)

CATEGORIES = """Categories:
- tasks (General): general things to do
- buy (Buy): things to purchase
- watch (Watch): movies, shows, videos to see
- ideas (Ideas): thoughts to remember
- work (Work): office and job tasks
- journal (Journal): feelings, emotions, stories with no action
"""   # exact training text
note = "call the bank at five and buy milk on the way home"
prompt = f"{CATEGORIES}Note: {note}\nJSON:\n"   # raw prompt, no chat template

ids = tok(prompt, return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=512, do_sample=False)  # greedy, as evaluated
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))
```

Weights are bf16. The duplicated tied `lm_head` was removed before upload (752,393,024 parameters). The architecture is `Qwen3_5ForCausalLM` (24 layers, hybrid Gated DeltaNet attention, vocabulary 248,320).

## Training

| Setting | Value |
|---|---|
| Run | EXP-031 `e2_lr1e-5_s0`, epoch 3 of 3 (seed 0) |
| Base | [`Qwen/Qwen3.5-0.8B-Base`](https://huggingface.co/Qwen/Qwen3.5-0.8B-Base), full fine-tune, 752,393,024 parameters |
| Data | 4,803 rows (see table) |
| Labels | Claude Sonnet 5.5 as the training label for the simulated notes (Sonnet and Gemini 3.1 Pro labelled them under guide v2); older notes keep their guide-v2 labels. No per-note human check |
| Recipe | lr 1e-5, AdamW, 5% warm-up then cosine, batch 4 × 4 = 16, max length 1,024, bf16 autocast, gradient checkpointing, 3 epochs, seed 0 |
| Steps / time | 903 steps, 22.52 minutes, best validation loss 0.1701 (step 301), peak 18.5 GB (batch 8 × 2 ran out of memory) |
| Hardware / cost | 1 × RTX 4090. About $0.34 of GPU for this run at the quoted $0.90 per hour (derived estimate, not an invoice) |

![Training curves](assets/training.png)

**The three data recipes:**

| Recipe | Rows | What it is |
|---|---|---|
| D3 (reference, EXP-030) | 3,388 | HeedSet 400 (Claude Opus, original labels) + 1,265 HeedSet labelled by Sonnet 5.5 + 777 notes I planned (generated with ChatGPT) relabelled by Sonnet 5.5 + 480 Claude brain dumps (220 Opus, 260 Sonnet) + 36 curated template notes + 140 real notes from me and five friends counted 3 times (420 rows) + 10 typed by me |
| E1 | 4,497 | D3 + all 1,109 simulated notes |
| **E2 (this model)** | **4,803** | E1 + one extra copy of the 306 "clear mistake" simulated notes |

The simulated notes were written by Claude Sonnet 5.5, Gemini 3.1 Pro and GPT-5.4 in rotation, with 10 of the real training notes shown as voice examples per call, then filtered for near-duplicates against validation, test and train. Before training, the D3 model answered all 1,109: 38.8% fully right. 306 were "clear mistakes" (the D3 answer differed from Sonnet's label while Gemini agreed with Sonnet) and 373 were unclear (the two teachers disagreed). Fully right by writer: Sonnet-written 48.0%, GPT-5.4-written 36.2%, Gemini-written 33.7%. The simulated notes were never used for evaluation. English only.

## Results

**Read this first.** The scores are **"fully right" shares against the older guide-v2 key**, on a 199-note yardstick (59 real notes + 140 HeedSet notes), scored by a judgement-free scorer. A note is fully right at **L3** when nothing is missing, extra, in the wrong bucket, or wrong in reminder or time. L2 leaves reminder and time out. They are **not comparable** with the word-overlap F1 of later models or with the blind-judge numbers on the [app model](https://huggingface.co/mohitkrconnect/heed-tasks-qwen3-0.6b) card. Intervals are cluster-bootstrap 95%.

### All runs of EXP-031

| Model | Epoch | Fully right L3 [95% CI] | Fully right L2 | Errors per note L2 / L3 | Synthetic slice (172) L3 |
|---|---|---|---|---|---|
| D3 seed 0 (EXP-030 reference, not published) | 3 | 47.2 | not recorded | 0.76 / 0.93 | 75.0 |
| E1 | 2 | 42.2 [35.2, 49.2] | 48.7 | 0.93 / 1.14 | 71.5 |
| E1 | 3 | 42.7 [35.7, 49.8] | 50.7 | 0.88 / 1.13 | 73.3 |
| E2 | 2 | 43.7 [36.7, 51.3] | 54.3 | 0.78 / 1.01 | 75.0 |
| **E2 (this model)** | **3** | **47.2** [40.2, 54.3] | 56.3 | 0.76 / 0.93 | 75.0 |
| E1 trained 5 epochs | 3 | 39.2 [32.2, 46.7] | 49.2 | 0.85 / 1.13 | 72.7 |
| E1 trained 5 epochs | 4 | 43.7 [37.2, 51.3] | 52.8 | 0.84 / 1.06 | 75.6 |
| E1 trained 5 epochs | 5 | 44.2 [37.7, 51.8] | 53.8 | 0.85 / 1.08 | 72.7 |
| D3 seed 1 (noise replicate) | 2 | 39.7 [33.2, 47.2] | 45.7 | 0.92 / 1.15 | 70.9 |
| D3 seed 1 | 3 | 44.2 [37.7, 51.3] | 50.7 | 0.83 / 1.03 | 75.0 |
| Soup of E2 ep3 + D3 seed 1 ep3 | n/a | 47.2 [40.7, 54.3] | 56.3 | 0.70 / 0.89 | not recorded |

Re-scoring the finalists one note at a time (the way a phone runs) gave identical answers.

**Pre-registered predictions:** P31.1 (E1 beats D3 by 1 to 5 points) false: E1 42.7 against 47.2. P31.2 (E2 within 2 of E1) false: +4.5. P31.9 (D3 seed 1 within 3 of 47.2) borderline: 44.2, 3.0 apart. P31.10 (batched against one-at-a-time within 2) true: identical.

## What it fed, and why it was not shipped

**D101: no measurable gain from 1,109 simulated notes, and a single-run gap under about 4 points is noise.** From then on recipes were judged on paired comparisons, per-field errors and at least two seeds. A failure analysis (Step 47) put about 55% of the gap to Sonnet in bucket errors.

It was not shipped because:
- it uses the wide format (intent, commands, reminder), and the project then narrowed to tasks only (D102);
- it is on Qwen3.5-0.8B, whose hybrid attention could not be run through Google's official LiteRT-LM converter (D103), so the project moved to Qwen3-0.6B;
- it was scored on the older yardstick, and later work moved to Sonnet-labelled F1 and a blind judge.

## Limitations

- A negative-result exhibit, not a product model.
- All numbers are on 199 notes (59 real), with intervals about 14 points wide, and carry seed noise of about 3 points.
- Model-written notes and Sonnet labels teach Claude's style.
- The 140 real notes appear three times in training.
- The model writes intent and command fields that the app does not use.
- Phone conversion, speed, memory, a judged score and a sealed-test score: not measured.
- Which label source (Sonnet silver or two-model merge) the 140 real notes carried inside D3, E1 and E2 was not confirmed.
- Memorisation of the real notes was not tested. English only.

## Links

- App and website: https://heed-xi.vercel.app
- Source code: https://github.com/t3chw/Heed_app
- App model: https://huggingface.co/mohitkrconnect/heed-tasks-qwen3-0.6b
- Base model: https://huggingface.co/Qwen/Qwen3.5-0.8B-Base

**Data:** the synthetic part of the training, validation and test notes, with all five systems' answers and the judge's verdicts on the public test notes, is in [mohitkrconnect/heed-voice-notes-to-tasks](https://huggingface.co/datasets/mohitkrconnect/heed-voice-notes-to-tasks). My real notes and my friends' are not published.

## License

Apache-2.0, the same as the base model. The training labels and most training notes were produced with Claude, Gemini and GPT models. Check those providers' terms before using this model to build a competing general-purpose model.