| --- |
| license: apache-2.0 |
| base_model: Qwen/Qwen2.5-0.5B-Instruct |
| language: |
| - en |
| pipeline_tag: text-generation |
| library_name: transformers |
| tags: |
| - information-extraction |
| - structured-output |
| - json |
| - maintenance |
| - onnx |
| - edge |
| --- |
| |
| # FIELDNOTEx-500M |
|
|
| Turns a technician's maintenance note into one JSON record that a |
| maintenance system can file. It is small enough to run on a Raspberry Pi 5 |
| with no network. |
|
|
| ``` |
| P-104 leaking at the seal, replaced 2x seal, pressure back to 3.2 bar, recheck in 2 weeks |
| ``` |
|
|
| ```json |
| {"asset_id": "P-104", "asset_type": "pump", "issue": "leak", "severity": null, |
| "action": "replaced_part", "parts": [{"part": "seal", "qty": 2}], |
| "reading": {"value": 3.2, "unit": "bar"}, "follow_up": true, "due_days": 14} |
| ``` |
|
|
| It is a full fine-tune of Qwen2.5-0.5B-Instruct. It comes as transformers |
| weights and as two ONNX exports, INT4 (452 MB) and INT8 (1,051 MB). |
|
|
| ## The record |
|
|
| | Field | Values | |
| |---|---| |
| | `asset_id` | the tag, upper case with a hyphen: `P-104`, `PR-15` | |
| | `asset_type` | pump, compressor, conveyor, motor, valve, boiler, chiller, fan, generator, press | |
| | `issue` | leak, vibration, overheating, noise, pressure_drop, electrical_fault, wear, blockage, corrosion, none | |
| | `severity` | low, medium, high, critical, or `null` when the note does not say | |
| | `action` | inspected, repaired, replaced_part, adjusted, cleaned, lubricated, shut_down, none | |
| | `parts` | `[{"part": "seal", "qty": 2}]`, or `[]` | |
| | `reading` | `{"value": 3.2, "unit": "bar"}` with unit C, bar, mm/s, A or rpm, or `null` | |
| | `follow_up` | `true` when the note asks for another visit | |
| | `due_days` | days until that visit ("2 weeks" is 14), or `null` | |
|
|
| The lists are fixed. The model has not learnt any value outside them, though |
| it can still write one (see below). Check every value against the lists before |
| you file the record. |
|
|
| ## Results |
|
|
| Exact match means every field of the record is right. The base model scores |
| 0% on both test sets. |
|
|
| | | Held-out phrasings (300) | Hand-written (40) | |
| |---|---|---| |
| | Full precision | 71.7% | 80.0% | |
| | ONNX INT8, 1,051 MB | 71.7% | 80.0% | |
| | ONNX INT4, 452 MB | 72.0% | 77.5% | |
|
|
| - **Held-out phrasings:** generated notes whose wording and layouts never |
| appear in training. They test whether the model learnt the task or the |
| sentences. |
| - **Hand-written:** 40 notes written by a person, outside any template. With 40 |
| cases, one note is 2.5 points. |
| - **INT8 against full precision:** the same answer on every one of the 340 |
| notes. |
| - **INT4 against full precision:** 5 notes broken and 6 mended on held-out, |
| and 1 broken on hand-written. That is within chance. |
|
|
| **Speed.** Decode speed on 4 threads of an x86 cloud host was 29 tok/s for INT4 |
| and 30 tok/s for INT8. A record is about 70 tokens, after a prompt of about |
| 270. **It has not been timed on a Raspberry Pi yet.** The memory-bandwidth |
| ceiling for a Pi 5 is about 45 tok/s for INT4 and 30 for INT8. |
|
|
| ## Use |
|
|
| The model was trained on one exact instruction, in `prompt.txt`, with the |
| note appended after `Field note: `. Use that prompt word for word and decode |
| greedily. Another prompt, or sampling, gives worse records. |
|
|
| ### transformers |
|
|
| ```python |
| import json |
| from huggingface_hub import hf_hub_download |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| repo = "thinkingdbx/FIELDNOTEx-500M" |
| prompt = open(hf_hub_download(repo, "prompt.txt")).read() |
| tok = AutoTokenizer.from_pretrained(repo) |
| model = AutoModelForCausalLM.from_pretrained(repo) |
| |
| note = "P-104 leaking at the seal, replaced 2x seal, pressure back to 3.2 bar, recheck in 2 weeks" |
| ids = tok.apply_chat_template([{"role": "user", "content": prompt + note}], |
| add_generation_prompt=True, return_tensors="pt", return_dict=True) |
| out = model.generate(**ids, max_new_tokens=200, do_sample=False) |
| record = json.loads(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True)) |
| ``` |
|
|
| ### ONNX Runtime (INT4 or INT8) |
|
|
| The exports were made with optimum-onnx 0.1.0 and onnxruntime 1.31.0. The |
| INT4 export uses `MatMulNBits`, which needs a recent ONNX Runtime. |
|
|
| ```python |
| import json |
| from huggingface_hub import snapshot_download |
| from optimum.onnxruntime import ORTModelForCausalLM |
| from transformers import AutoTokenizer |
| |
| path = snapshot_download("thinkingdbx/FIELDNOTEx-500M", allow_patterns=["onnx/int4/*", "prompt.txt"]) |
| prompt = open(f"{path}/prompt.txt").read() |
| tok = AutoTokenizer.from_pretrained(f"{path}/onnx/int4") |
| model = ORTModelForCausalLM.from_pretrained(f"{path}/onnx/int4") |
| |
| note = "compressor C-7 overheating again, 104C at the discharge. shut it down until we get parts. urgent - critical" |
| ids = tok.apply_chat_template([{"role": "user", "content": prompt + note}], |
| add_generation_prompt=True, return_tensors="pt", return_dict=True) |
| out = model.generate(**ids, max_new_tokens=200, do_sample=False) |
| record = json.loads(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True)) |
| ``` |
|
|
| Every one of the 340 test answers parsed as JSON. Still validate the record, |
| values included, before you file it. |
|
|
| ## Files |
|
|
| | Path | What | Size | |
| |---|---|---| |
| | `model.safetensors` | full-precision weights (fp32) | 1.98 GB | |
| | `onnx/int4/` | 4-bit weights in blocks of 16, symmetric; embedding table quantised too | 452 MB | |
| | `onnx/int8/` | 8-bit weight-only in blocks of 32; embedding table kept at fp32 | 1,051 MB | |
| | `prompt.txt` | the instruction the model was trained on | | |
|
|
| ## What it gets wrong |
|
|
| Most misses on held-out notes are in `issue` (43–46 of 300 notes) and `action` |
| (26). They cluster on wording the model never saw: |
|
|
| - **"sorted it out"** comes out as `"action": "sorted"`, which is not a value |
| in the list, on all 26 held-out notes. It should be `repaired`. Words copied |
| from the note are the commonest way the model leaves its lists. |
| - **"running rough"** gives `vibration` in 4 notes of 16, and `none` in the |
| other 12. |
| - **"keeps tripping out"** (an electrical fault) is read as `vibration` in 33 |
| held-out notes of 34, and on the hand-written note too. |
| - **Counts in words**, as in "replaced the seals, 3 of them", can come out as |
| `qty: 1`. |
| - **The asset tag is not used** to infer the asset type. A note that says only |
| "F-9" with no other word for fan relies on the wording. |
|
|
| How well the model reads one particular phrasing also moved between two |
| training runs that differed only in the random draw of training notes. Treat |
| any single phrasing in the lists above as an example, not a guarantee. |
|
|
| ## Training |
|
|
| - **Data:** 8,000 synthetic notes from a deterministic generator. The |
| generator computes every label. A language model never wrote or checked a |
| record. |
| - **Phrasing:** each field drew its wording from pools of hand-written |
| phrasings, 6 to 8 per value. On top of those, Qwen2.5-7B-Instruct proposed |
| 364 phrasings, which a person reviewed one by one. Any proposed phrasing |
| that appeared in a test note, or contained one, was removed before review. |
| - **Run:** full fine-tune, 500 steps at an effective batch of 32, learning rate |
| 2e-5 with cosine decay, sequence length 512, on one NVIDIA L4. |
| - **Exports:** made after training, then scored again on both test sets with |
| ONNX Runtime. An export would have been refused if it had lost more than 2 |
| points on either set, with more answers broken than chance allows. |
|
|
| Trained with Foundry by thinkingdbx. |
|
|
| ## Limitations |
|
|
| - **English only.** The training notes use industrial maintenance language, |
| and only that. |
| - **One note, one record.** A note covering two assets gets one record. |
| - **Synthetic training data.** The hand-written set is the best guide to real |
| notes, and it has only 40 notes. Test it on your own notes before relying on |
| it. |
| - **Fixed lists.** Assets, issues and units outside the table are not |
| covered. Expect a wrong value, or a word copied from the note. |
|
|
| ## License |
|
|
| Apache 2.0, as for the base model, Qwen2.5-0.5B-Instruct. |
|
|
