docs/LLM_LABELLING_PLAN.md

# v2 plan — LLM-assisted labelling, data expansion, and synthetic generation

Status: **design, not yet built.** v1 (ensemble + document-level CV + honest eval)
ships first. This document specifies the v2 data work so it can be executed with
the rigour it needs rather than rushed.

The diagnosis from v1 is that the task is **data-limited, not
architecture-limited**: training loss reaches 0.01 while dev macro-F1 stalls at
~0.41, and the model covers only **49 of ~697** ATT&CK Enterprise techniques
because only 49 clear the ≥3-documents-for-a-leak-free-split bar. So the highest
lever is *more and broader labelled data*, obtained without inflating the
evaluation.

A 50-sentence blind pilot (one Claude pass, no tooling) scored **micro-F1 0.705
vs human gold**, and on inspection **~25% of the disagreements were human
annotation errors** — the labeller both adds data and surfaces gold noise. That
is the evidence base for what follows.

---

## The one non-negotiable rule

> **Machine labels (pseudo-labelled or synthetic) go only into TRAIN.
> Dev and test remain 100% human gold, forever.**

This keeps every headline number a real, human-validated measurement. Machine
labels become a *training augmentation whose value is measured on the honest
test set*, never part of the target. Break this rule and the whole contribution
collapses into circular self-grading.

Every row carries a `label_provenance` field: `human` | `procedure_example` |
`pseudo_llm` | `synthetic_llm`. Nothing is anonymous.

---

## Three data-expansion strategies (different tools for different gaps)

| # | Strategy | Fixes | Text realism | Risk |
|---|---|---|---|---|
| A | **Pseudo-label real unlabelled TRAM sentences** | depth for the 49; recovers annotator misses | real prose (ideal) | low — bounded by labeller precision |
| B | **Harvest ATT&CK procedure examples** | breadth (covers 608 techniques) + depth | real, but terse/templated style | medium — distribution shift from report prose |
| C | **Synthetic generation** | breadth for **zero-data** techniques only | model-written (weakest) | high — style artifacts, homogeneity, leakage |

Apply in that order of preference. Reach for C only where A and B are empty (a
technique with no real sentences anywhere), because that is the only place its
risk is justified.

---

## The labelling prompt

Engineered directly from the pilot's failure modes. Each rule below maps to a
mistake that was actually observed (human or model).

### System / instruction block

```
You are an expert MITRE ATT&CK analyst labelling sentences from public cyber
threat-intelligence reports. For each sentence, identify every ATT&CK technique
whose BEHAVIOUR IS ACTUALLY DESCRIBED AS OCCURRING in that sentence.

LABEL SPACE
- Choose only from the CANDIDATE TECHNIQUES list provided below. Each entry is
  "ID  Name — one-line definition".
- A sentence may map to zero, one, or several techniques (multi-label).
- If nothing in the list is genuinely evidenced, return an empty list. Empty is
  a valid, common, and correct answer. Do NOT force a label.

WHAT COUNTS AS EVIDENCE (tag it)
- An adversary/malware ACTION that instantiates a technique
  ("dumped credentials from LSASS" -> T1003.001).
- Capture ALL techniques present. "opened the HTML attachment from the phishing
  email" is BOTH T1566.001 (spearphishing attachment) AND T1204.002 (user
  execution) — and if the attachment contained "junk/HEX-encoded data",
  ALSO T1027 (obfuscation). Do not stop at the first match.
- If the report text literally names a technique or its ID, trust that as strong
  evidence — but still verify it fits the sentence's described behaviour.

WHAT DOES NOT COUNT (return empty / do not tag)
- Analyst or researcher actions, not the adversary's:
  "running the file through a debugger and dumping memory for analysis" is the
  RESEARCHER reverse-engineering, not an ATT&CK technique.
- Pure indicators, config dumps, hashes, IPs, domains, figure captions,
  file-path listings, or meta-commentary ("more information can be found here",
  "they probably bought the code on a market"). These are NONE.
- A capability described in the abstract with no actor performing it, UNLESS the
  sentence is clearly attributing it to the malware/actor.

DISAMBIGUATION (the confusable pairs that trip both humans and models)
- T1027 Obfuscated Files or Information  vs  T1140 Deobfuscate/Decode:
  T1027 = the data is MADE obfuscated/encoded/packed. T1140 = the malware
  DECODES/DECRYPTS/UNPACKS at runtime. "saved as Base64" -> T1027.
  "decodes the payload and runs it" -> T1140 (and often T1027 too if it says
  the payload was encoded). When both the encoding and the decoding are
  described, tag both.
- T1055 Process Injection  vs  T1106 Native API:
  Injection into another process -> T1055. Calling OS APIs / raw syscalls as the
  mechanism -> T1106. "injects via raw NTDLL syscalls" legitimately implies
  BOTH; prefer tagging both when both are explicit.
- T1070.004 File Deletion  vs  T1027 Obfuscation:
  Deleting files to cover tracks -> T1070.004. Do not mislabel deletion as
  obfuscation.
- OS SCOPE MATTERS for sub-techniques. T1059.003 is specifically the *Windows*
  Command Shell (cmd.exe/.bat). A "/bin/sh subprocess" is a Unix shell and is
  NOT T1059.003; if the Unix variant is not in the candidate list, return empty
  rather than forcing the Windows one.
- T1005 Data from Local System  vs  network shares: reviewing documents on
  network shares is not local-system collection; if only the network-share
  technique fits and it is absent from the list, return empty.

OUTPUT — strict JSON, one object per sentence:
{
  "id": <int>,
  "techniques": [
    {"id": "T1003.001",
     "confidence": "high|medium|low",
     "evidence": "<exact 3-12 word span from the sentence that triggered it>",
     "why": "<max 15 words>"}
  ]
}
Rules for output:
- "evidence" MUST be a verbatim substring of the sentence. If you cannot quote a
  trigger span, do not emit the label.
- Use "high" only when the behaviour is explicit and unambiguous. Use "low" when
  you are inferring. Downstream keeps only "high" (and, for rare techniques,
  "high"+"medium") — so err toward honest, lower confidence rather than
  overclaiming.
- Never invent an ID that is not in the candidate list.
```

### Why each rule exists (pilot trace)

| Rule | Pilot case it fixes |
|---|---|
| "analyst actions are not techniques" | #19 debugger/memory-dump was researcher activity |
| "indicators/filler are NONE" | #40 filler sentence had been (wrongly) gold-tagged T1090 |
| "capture ALL techniques" | #8 model missed T1027 alongside phishing/user-execution |
| T1027/T1140 split | #31, #34, #41 sibling confusion (both humans and model) |
| T1055/T1106 split | #32 injection-vs-native-API |
| T1070.004 vs T1027 | #49 deletion mislabelled as obfuscation in gold |
| OS scope | #20 `/bin/sh` wrongly forced to a Windows label |
| require verbatim `evidence` span | hallucination guard; makes every label auditable |
| confidence + abstention | lets us trade recall for precision above the pilot's 0.78 |

---

## Validation protocol (this is what makes it a contribution, not noise)

1. **Blind gold audit (calibration).** Run the labeller on a stratified 300–500
   sentence sample of the *human gold* dev set, blind to the gold. Compute
   agreement. Then adjudicate every disagreement into
   `{llm-correct, gold-correct, both-defensible}` — this yields (a) the
   labeller's true precision/recall against *clean* gold, and (b) a **quantified
   estimate of gold-label noise**, which is itself a headline finding and a
   ceiling on any model's achievable score.

2. **Confidence calibration.** From the audit, pick the confidence threshold
   that hits a target precision (e.g. ≥0.90) on held-out gold. Only labels at/above
   that threshold enter TRAIN.

3. **Self-consistency.** Run 2–3 independent passes (prompt-order shuffle and/or
   modest temperature). Keep a positive label only where passes agree. Agreement
   rate is logged per technique.

4. **Downstream ablation (the actual payoff).** Retrain the v1 best config on
   `gold` vs `gold + pseudo` vs `gold + pseudo + procedure`, and report the
   leak-free **document-level CV** macro-F1 with error bars for each. The claim
   is only ever "adding source X moved the honest test score by Δ ± σ". Test and
   dev stay pure human gold throughout.

5. **Coverage report.** Track how many of the 697 techniques become
   trainable-and-evaluable as data is added — the breadth story.

### Go / no-go gates

- Gate 1 (after audit): if calibrated precision at the chosen threshold is
  < ~0.85 on clean gold, **stop** — the labeller is not reliable enough; ship v1
  and reconsider. The pilot's uncalibrated 0.78 suggests calibration clears this,
  but it must be shown, not assumed.
- Gate 2 (after ablation): if `gold+pseudo` does not beat `gold` on the honest
  CV outside error bars, publish the **negative** result (data expansion via LLM
  did not help this task) — still a legitimate, useful finding. Do not bury it.

---

## Synthetic data (strategy C) — yes, but tightly fenced

Useful specifically for the **breadth** problem: techniques with zero or near-zero
real sentences, where pseudo-labelling has nothing to label. There, synthetic
examples are the only way to make a technique trainable at all.

### Method
- Condition generation on the ATT&CK **technique description + real procedure
  examples** for that technique. Ask for diverse, realistic *report-style*
  sentences: vary actor, tool, OS, phrasing, sentence position; mix in benign
  context so the model cannot key on a template.
- Generate a surplus, then **filter**: drop any sentence that names the technique
  ID or its exact title (that would leak the label into the input), de-duplicate
  against all real data, and drop near-duplicates of each other.

### Guardrails (non-negotiable)
- Synthetic rows are **TRAIN-only**, `label_provenance: synthetic_llm`, and
  capped as a fraction of any technique's training pool.
- A **real-only held-out test** is the sole judge. If the model only improves on
  synthetic-flavoured text and not on real report prose, the synthetic data is
  teaching style artifacts — cut it.
- Human spot-check a sample of every generated technique for plausibility.
- Report results **with and without** synthetic data separately, so a reader can
  discount it if they wish.

### The honest risk statement (goes in the dataset card)
Synthetic sentences are model-written and may encode the generator's blind spots
and stylistic tells. They exist to bootstrap zero-data techniques, are confined
to training, are never used to compute any reported score, and their marginal
effect is reported as an isolated ablation.

---

## Rough effort / cost

- Procedure-example harvest (strategy B): ~1 day, no API cost (already on disk).
- Pseudo-labelling ~14k sentences (strategy A): batched API calls (10–20
  sentences/call), a few dollars to low tens of dollars depending on model and
  passes; ~1–2 days including the audit.
- Synthetic (strategy C): scoped per target technique; only for the zero-data
  tail; ~1–2 days.
- All retraining fits on the existing RTX 4060 — no rental.

## Order of work

1. Ship v1 (ensemble + CV + honest cards).
2. Strategy B (procedure examples) — cheapest, real text, biggest breadth gain.
3. Strategy A (pseudo-label) with full validation protocol.
4. Strategy C (synthetic) only for the residual zero-data tail.
5. Release as a **v2 dataset** with provenance-tagged rows, the gold-noise audit,
   and the ablation table — keeping v1 intact and reproducible.
```