v2 plan — LLM-assisted labelling, data expansion, and synthetic generation

Status: design, not yet built. v1 (ensemble + document-level CV + honest eval) ships first. This document specifies the v2 data work so it can be executed with the rigour it needs rather than rushed.

The diagnosis from v1 is that the task is data-limited, not architecture-limited: training loss reaches 0.01 while dev macro-F1 stalls at ~0.41, and the model covers only 49 of ~697 ATT&CK Enterprise techniques because only 49 clear the ≥3-documents-for-a-leak-free-split bar. So the highest lever is more and broader labelled data, obtained without inflating the evaluation.

A 50-sentence blind pilot (one Claude pass, no tooling) scored micro-F1 0.705 vs human gold, and on inspection ~25% of the disagreements were human annotation errors — the labeller both adds data and surfaces gold noise. That is the evidence base for what follows.

The one non-negotiable rule

Machine labels (pseudo-labelled or synthetic) go only into TRAIN. Dev and test remain 100% human gold, forever.

This keeps every headline number a real, human-validated measurement. Machine labels become a training augmentation whose value is measured on the honest test set, never part of the target. Break this rule and the whole contribution collapses into circular self-grading.

Every row carries a label_provenance field: human | procedure_example | pseudo_llm | synthetic_llm. Nothing is anonymous.

Three data-expansion strategies (different tools for different gaps)

#	Strategy	Fixes	Text realism	Risk
A	Pseudo-label real unlabelled TRAM sentences	depth for the 49; recovers annotator misses	real prose (ideal)	low — bounded by labeller precision
B	Harvest ATT&CK procedure examples	breadth (covers 608 techniques) + depth	real, but terse/templated style	medium — distribution shift from report prose
C	Synthetic generation	breadth for zero-data techniques only	model-written (weakest)	high — style artifacts, homogeneity, leakage

Apply in that order of preference. Reach for C only where A and B are empty (a technique with no real sentences anywhere), because that is the only place its risk is justified.

The labelling prompt

Engineered directly from the pilot's failure modes. Each rule below maps to a mistake that was actually observed (human or model).

System / instruction block

You are an expert MITRE ATT&CK analyst labelling sentences from public cyber
threat-intelligence reports. For each sentence, identify every ATT&CK technique
whose BEHAVIOUR IS ACTUALLY DESCRIBED AS OCCURRING in that sentence.

LABEL SPACE
- Choose only from the CANDIDATE TECHNIQUES list provided below. Each entry is
  "ID  Name — one-line definition".
- A sentence may map to zero, one, or several techniques (multi-label).
- If nothing in the list is genuinely evidenced, return an empty list. Empty is
  a valid, common, and correct answer. Do NOT force a label.

WHAT COUNTS AS EVIDENCE (tag it)
- An adversary/malware ACTION that instantiates a technique
  ("dumped credentials from LSASS" -> T1003.001).
- Capture ALL techniques present. "opened the HTML attachment from the phishing
  email" is BOTH T1566.001 (spearphishing attachment) AND T1204.002 (user
  execution) — and if the attachment contained "junk/HEX-encoded data",
  ALSO T1027 (obfuscation). Do not stop at the first match.
- If the report text literally names a technique or its ID, trust that as strong
  evidence — but still verify it fits the sentence's described behaviour.

WHAT DOES NOT COUNT (return empty / do not tag)
- Analyst or researcher actions, not the adversary's:
  "running the file through a debugger and dumping memory for analysis" is the
  RESEARCHER reverse-engineering, not an ATT&CK technique.
- Pure indicators, config dumps, hashes, IPs, domains, figure captions,
  file-path listings, or meta-commentary ("more information can be found here",
  "they probably bought the code on a market"). These are NONE.
- A capability described in the abstract with no actor performing it, UNLESS the
  sentence is clearly attributing it to the malware/actor.

DISAMBIGUATION (the confusable pairs that trip both humans and models)
- T1027 Obfuscated Files or Information  vs  T1140 Deobfuscate/Decode:
  T1027 = the data is MADE obfuscated/encoded/packed. T1140 = the malware
  DECODES/DECRYPTS/UNPACKS at runtime. "saved as Base64" -> T1027.
  "decodes the payload and runs it" -> T1140 (and often T1027 too if it says
  the payload was encoded). When both the encoding and the decoding are
  described, tag both.
- T1055 Process Injection  vs  T1106 Native API:
  Injection into another process -> T1055. Calling OS APIs / raw syscalls as the
  mechanism -> T1106. "injects via raw NTDLL syscalls" legitimately implies
  BOTH; prefer tagging both when both are explicit.
- T1070.004 File Deletion  vs  T1027 Obfuscation:
  Deleting files to cover tracks -> T1070.004. Do not mislabel deletion as
  obfuscation.
- OS SCOPE MATTERS for sub-techniques. T1059.003 is specifically the *Windows*
  Command Shell (cmd.exe/.bat). A "/bin/sh subprocess" is a Unix shell and is
  NOT T1059.003; if the Unix variant is not in the candidate list, return empty
  rather than forcing the Windows one.
- T1005 Data from Local System  vs  network shares: reviewing documents on
  network shares is not local-system collection; if only the network-share
  technique fits and it is absent from the list, return empty.

OUTPUT — strict JSON, one object per sentence:
{
  "id": <int>,
  "techniques": [
    {"id": "T1003.001",
     "confidence": "high|medium|low",
     "evidence": "<exact 3-12 word span from the sentence that triggered it>",
     "why": "<max 15 words>"}
  ]
}
Rules for output:
- "evidence" MUST be a verbatim substring of the sentence. If you cannot quote a
  trigger span, do not emit the label.
- Use "high" only when the behaviour is explicit and unambiguous. Use "low" when
  you are inferring. Downstream keeps only "high" (and, for rare techniques,
  "high"+"medium") — so err toward honest, lower confidence rather than
  overclaiming.
- Never invent an ID that is not in the candidate list.

Why each rule exists (pilot trace)

Rule	Pilot case it fixes
"analyst actions are not techniques"	#19 debugger/memory-dump was researcher activity
"indicators/filler are NONE"	#40 filler sentence had been (wrongly) gold-tagged T1090
"capture ALL techniques"	#8 model missed T1027 alongside phishing/user-execution
T1027/T1140 split	#31, #34, #41 sibling confusion (both humans and model)
T1055/T1106 split	#32 injection-vs-native-API
T1070.004 vs T1027	#49 deletion mislabelled as obfuscation in gold
OS scope	#20 `/bin/sh` wrongly forced to a Windows label
require verbatim `evidence` span	hallucination guard; makes every label auditable
confidence + abstention	lets us trade recall for precision above the pilot's 0.78

Validation protocol (this is what makes it a contribution, not noise)

Blind gold audit (calibration). Run the labeller on a stratified 300–500 sentence sample of the human gold dev set, blind to the gold. Compute agreement. Then adjudicate every disagreement into {llm-correct, gold-correct, both-defensible} — this yields (a) the labeller's true precision/recall against clean gold, and (b) a quantified estimate of gold-label noise, which is itself a headline finding and a ceiling on any model's achievable score.
Confidence calibration. From the audit, pick the confidence threshold that hits a target precision (e.g. ≥0.90) on held-out gold. Only labels at/above that threshold enter TRAIN.
Self-consistency. Run 2–3 independent passes (prompt-order shuffle and/or modest temperature). Keep a positive label only where passes agree. Agreement rate is logged per technique.
Downstream ablation (the actual payoff). Retrain the v1 best config on gold vs gold + pseudo vs gold + pseudo + procedure, and report the leak-free document-level CV macro-F1 with error bars for each. The claim is only ever "adding source X moved the honest test score by Δ ± σ". Test and dev stay pure human gold throughout.
Coverage report. Track how many of the 697 techniques become trainable-and-evaluable as data is added — the breadth story.

Go / no-go gates

Gate 1 (after audit): if calibrated precision at the chosen threshold is < ~0.85 on clean gold, stop — the labeller is not reliable enough; ship v1 and reconsider. The pilot's uncalibrated 0.78 suggests calibration clears this, but it must be shown, not assumed.
Gate 2 (after ablation): if gold+pseudo does not beat gold on the honest CV outside error bars, publish the negative result (data expansion via LLM did not help this task) — still a legitimate, useful finding. Do not bury it.

Synthetic data (strategy C) — yes, but tightly fenced

Useful specifically for the breadth problem: techniques with zero or near-zero real sentences, where pseudo-labelling has nothing to label. There, synthetic examples are the only way to make a technique trainable at all.

Method

Condition generation on the ATT&CK technique description + real procedure examples for that technique. Ask for diverse, realistic report-style sentences: vary actor, tool, OS, phrasing, sentence position; mix in benign context so the model cannot key on a template.
Generate a surplus, then filter: drop any sentence that names the technique ID or its exact title (that would leak the label into the input), de-duplicate against all real data, and drop near-duplicates of each other.

Guardrails (non-negotiable)

Synthetic rows are TRAIN-only, label_provenance: synthetic_llm, and capped as a fraction of any technique's training pool.
A real-only held-out test is the sole judge. If the model only improves on synthetic-flavoured text and not on real report prose, the synthetic data is teaching style artifacts — cut it.
Human spot-check a sample of every generated technique for plausibility.
Report results with and without synthetic data separately, so a reader can discount it if they wish.

The honest risk statement (goes in the dataset card)

Synthetic sentences are model-written and may encode the generator's blind spots and stylistic tells. They exist to bootstrap zero-data techniques, are confined to training, are never used to compute any reported score, and their marginal effect is reported as an isolated ablation.

Rough effort / cost

Procedure-example harvest (strategy B): ~1 day, no API cost (already on disk).
Pseudo-labelling ~14k sentences (strategy A): batched API calls (10–20 sentences/call), a few dollars to low tens of dollars depending on model and passes; ~1–2 days including the audit.
Synthetic (strategy C): scoped per target technique; only for the zero-data tail; ~1–2 days.
All retraining fits on the existing RTX 4060 — no rental.

Order of work

Ship v1 (ensemble + CV + honest cards).
Strategy B (procedure examples) — cheapest, real text, biggest breadth gain.
Strategy A (pseudo-label) with full validation protocol.
Strategy C (synthetic) only for the residual zero-data tail.
Release as a v2 dataset with provenance-tagged rows, the gold-noise audit, and the ablation table — keeping v1 intact and reproducible.