tiny-ncm weights (task6), tokenizer, tool registry, model card
Browse files- README.md +95 -0
- bpe.json +0 -0
- registry.json +24 -0
- tiny-ncm.pt +3 -0
README.md
ADDED
|
@@ -0,0 +1,95 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: cc-by-nc-4.0
|
| 3 |
+
language:
|
| 4 |
+
- en
|
| 5 |
+
library_name: pytorch
|
| 6 |
+
pipeline_tag: text-generation
|
| 7 |
+
tags:
|
| 8 |
+
- ncm
|
| 9 |
+
- coarse-to-fine
|
| 10 |
+
- vq-vae
|
| 11 |
+
- tool-use
|
| 12 |
+
- from-scratch
|
| 13 |
+
- tiny
|
| 14 |
+
datasets:
|
| 15 |
+
- HuggingFaceFW/fineweb-edu
|
| 16 |
+
- roneneldan/TinyStories
|
| 17 |
+
- wikimedia/wikipedia
|
| 18 |
+
- abisee/cnn_dailymail
|
| 19 |
+
- corbt/enron-emails
|
| 20 |
+
- Yale-LILY/aeslc
|
| 21 |
+
- HuggingFaceTB/everyday-conversations-llama3.1-2k
|
| 22 |
+
- HuggingFaceTB/smol-smoltalk
|
| 23 |
+
- Salesforce/xlam-function-calling-60k
|
| 24 |
+
---
|
| 25 |
+
|
| 26 |
+
**Files:** `tiny-ncm.pt` (weights + config, 146 MB), `bpe.json` (text tokenizer), `registry.json` (tool list).
|
| 27 |
+
Run it with the code at https://github.com/amar683/tiny-ncm:
|
| 28 |
+
|
| 29 |
+
```bash
|
| 30 |
+
git clone https://github.com/amar683/tiny-ncm && cd tiny-ncm
|
| 31 |
+
python3 -m venv .venv && .venv/bin/pip install -r requirements.txt
|
| 32 |
+
hf download amartomar683/tiny-ncm tiny-ncm.pt --local-dir weights
|
| 33 |
+
.venv/bin/python chat.py --dry-run
|
| 34 |
+
```
|
| 35 |
+
|
| 36 |
+
# Model card: tiny-ncm
|
| 37 |
+
|
| 38 |
+
## Summary
|
| 39 |
+
- **Type:** NCM (coarse-to-fine, block-wise) language model for tool use, trained from scratch.
|
| 40 |
+
- **Size:** 36.4M parameters: generator 20.9M (d=384, 8 layers, 3 query heads / 1 KV head) and multiscale residual VQ
|
| 41 |
+
tokenizer 15.5M (5 levels, 1024-entry codebooks). Byte-level BPE vocabulary of 16,384. Context 4096.
|
| 42 |
+
- **Checkpoint:** `runs/task6` (released as `tiny-ncm.pt`, fp32, 146 MB). Code: https://github.com/amar683/tiny-ncm (MIT).
|
| 43 |
+
- **Hardware:** one MacBook (Apple M5, 24 GB), PyTorch MPS, bf16. About 12 hours of training in total (VQ tokenizer 1.3 h, generator 10.5 h).
|
| 44 |
+
|
| 45 |
+
## Intended use
|
| 46 |
+
A personal assistant that turns everyday requests into tool calls (alarms, reminders, timers, calendar, read-only Gmail,
|
| 47 |
+
web and Wikipedia lookups, weather, notes, calculator) and makes small talk. Use it together with the app in
|
| 48 |
+
`assistant/`, which checks every tool call against the user's words. **Not** for coding, reasoning, medical, legal or
|
| 49 |
+
financial advice, or anything where a wrong answer matters.
|
| 50 |
+
|
| 51 |
+
## Training data
|
| 52 |
+
| Data | License | Used for |
|
| 53 |
+
|---|---|---|
|
| 54 |
+
| [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) (sample-10BT) | ODC-By 1.0 | general English |
|
| 55 |
+
| [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories) | CDLA-Sharing 1.0 | simple English |
|
| 56 |
+
| [Wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) (20231101.en) | CC BY-SA 3.0 / GFDL | text, lookup answers |
|
| 57 |
+
| [CNN/DailyMail](https://huggingface.co/datasets/abisee/cnn_dailymail) | Apache-2.0 | summaries |
|
| 58 |
+
| [Enron emails](https://huggingface.co/datasets/corbt/enron-emails), [AESLC](https://huggingface.co/datasets/Yale-LILY/aeslc) | not stated (public record) | email style, subject lines |
|
| 59 |
+
| [everyday-conversations](https://huggingface.co/datasets/HuggingFaceTB/everyday-conversations-llama3.1-2k), [smol-smoltalk](https://huggingface.co/datasets/HuggingFaceTB/smol-smoltalk) | Apache-2.0 | small talk (non-coding only) |
|
| 60 |
+
| [xLAM function-calling 60k](https://huggingface.co/datasets/Salesforce/xlam-function-calling-60k) | CC-BY-4.0, "research purposes only" | reading tool schemas |
|
| 61 |
+
| Synthetic tool conversations (`ncm/synth_tools.py`, `make_copy_data.py`) | made for this project | tool calls |
|
| 62 |
+
|
| 63 |
+
The weights were **not** trained on web pages collected by `harvest_web.py` (that experiment, `runs/web1`, was not adopted).
|
| 64 |
+
|
| 65 |
+
**Weights license: CC BY-NC 4.0.** Because of the xLAM terms and the data with no stated license, the weights are released for
|
| 66 |
+
**research and non-commercial use**, with credit to the datasets above. xLAM asks users to cite APIGen:
|
| 67 |
+
Liu et al., *APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets*, 2024.
|
| 68 |
+
|
| 69 |
+
## Training
|
| 70 |
+
1. BPE tokenizer (16k) on the mix.
|
| 71 |
+
2. VQ tokenizer: 97.6% exact text reconstruction, 95% exact tool JSON.
|
| 72 |
+
3. Generator pretraining at context 1024 (Muon + AdamW, warmup-stable-decay schedule).
|
| 73 |
+
4. Chat/tool phases at 4096: synthetic tool data, then "copy-focused" data with fresh random names, times and places in every
|
| 74 |
+
example, then 2h45m of mostly English, then 2 × 1h of copy-focused data.
|
| 75 |
+
Details and every run: [docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md).
|
| 76 |
+
|
| 77 |
+
## Evaluation
|
| 78 |
+
| Test | Model + app checks | Model alone |
|
| 79 |
+
|---|---|---|
|
| 80 |
+
| 500 new requests | 490 / 500 | n/a |
|
| 81 |
+
| 60 hard requests written before the fixes | 57 / 60 | n/a |
|
| 82 |
+
| 153 everyday requests, right tool | 153 / 153 | n/a |
|
| 83 |
+
| 153 everyday requests, every detail exact | 153 / 153 | 64 / 153 |
|
| 84 |
+
| Math (90) | 89 / 90 | n/a |
|
| 85 |
+
| Coding (40) | 5 / 40 | n/a |
|
| 86 |
+
|
| 87 |
+
## Limitations and risks
|
| 88 |
+
- **Copying.** The model often garbles names, numbers and times ("London" → "Singapore"); the app fixes them from the
|
| 89 |
+
user's words. Without the app, expect about half of the tool calls to have a wrong detail.
|
| 90 |
+
- **English.** Free-form replies are rough; the app replaces replies that don't match the tool result.
|
| 91 |
+
- **Reasoning.** No multi-step math, no code.
|
| 92 |
+
- **Web answers** are picked from the top search results by word matching; they can be wrong or mix methods.
|
| 93 |
+
- **No safety training.** The model has no refusal or content filtering. The app asks before any action that sends
|
| 94 |
+
messages or calls someone, and never guesses those tools from a garbled name.
|
| 95 |
+
- **Privacy.** Everything runs locally. Web questions go to DuckDuckGo and the pages' sites; Gmail access is read-only.
|
bpe.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
registry.json
ADDED
|
@@ -0,0 +1,24 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"version": 1,
|
| 3 |
+
"tools": [
|
| 4 |
+
{"name": "set_alarm", "desc": "set an alarm", "args": {"time": "str", "label": "str?"}, "platforms": ["mac", "windows", "linux", "android", "ios"]},
|
| 5 |
+
{"name": "set_reminder", "desc": "create a reminder", "args": {"text": "str", "time": "str?"}, "platforms": ["mac", "windows", "linux", "android", "ios"]},
|
| 6 |
+
{"name": "set_timer", "desc": "start a countdown timer", "args": {"duration": "str", "label": "str?"}, "platforms": ["mac", "windows", "linux", "android", "ios"]},
|
| 7 |
+
{"name": "calendar_add", "desc": "add a calendar event", "args": {"title": "str", "time": "str", "duration": "str?"}, "platforms": ["mac", "android", "ios"]},
|
| 8 |
+
{"name": "calendar_list", "desc": "list calendar events", "args": {"day": "str"}, "platforms": ["mac", "android", "ios"]},
|
| 9 |
+
{"name": "list_emails", "desc": "list recent emails", "args": {"query": "str?", "n": "int?"}, "platforms": ["mac", "windows", "linux", "android", "ios"]},
|
| 10 |
+
{"name": "read_email", "desc": "read one email", "args": {"id": "str"}, "platforms": ["mac", "windows", "linux", "android", "ios"]},
|
| 11 |
+
{"name": "draft_email", "desc": "save an email draft", "args": {"to": "str", "subject": "str", "body": "str"}, "platforms": ["mac", "windows", "linux", "android", "ios"]},
|
| 12 |
+
{"name": "send_email", "desc": "send an email", "args": {"to": "str", "subject": "str", "body": "str"}, "platforms": ["mac", "windows", "linux", "android", "ios"], "confirm": true},
|
| 13 |
+
{"name": "lookup", "desc": "look up a topic in an encyclopedia", "args": {"topic": "str"}, "platforms": ["mac", "windows", "linux", "android", "ios"]},
|
| 14 |
+
{"name": "web_search", "desc": "search the web", "args": {"query": "str"}, "platforms": ["mac", "windows", "linux", "android", "ios"]},
|
| 15 |
+
{"name": "weather", "desc": "get the weather", "args": {"location": "str?", "when": "str?"}, "platforms": ["mac", "windows", "linux", "android", "ios"]},
|
| 16 |
+
{"name": "notes_add", "desc": "save a note", "args": {"text": "str"}, "platforms": ["mac", "windows", "linux", "android", "ios"]},
|
| 17 |
+
{"name": "notes_search", "desc": "search saved notes", "args": {"query": "str"}, "platforms": ["mac", "windows", "linux", "android", "ios"]},
|
| 18 |
+
{"name": "calculator", "desc": "evaluate arithmetic", "args": {"expression": "str"}, "platforms": ["mac", "windows", "linux", "android", "ios"]},
|
| 19 |
+
{"name": "get_time", "desc": "current time in a place", "args": {"place": "str"}, "platforms": ["mac", "windows", "linux", "android", "ios"]},
|
| 20 |
+
{"name": "open_app", "desc": "open an app", "args": {"app": "str"}, "platforms": ["mac", "windows", "android", "ios"]},
|
| 21 |
+
{"name": "call", "desc": "phone call a contact", "args": {"contact": "str"}, "platforms": ["android", "ios"], "confirm": true},
|
| 22 |
+
{"name": "sms", "desc": "send a text message", "args": {"to": "str", "message": "str"}, "platforms": ["android", "ios"], "confirm": true}
|
| 23 |
+
]
|
| 24 |
+
}
|
tiny-ncm.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:085a893255dbe9e946f5099d8b4c6943cbd63f8e83b4c270e0b7b78b90fcaec7
|
| 3 |
+
size 145837082
|