rigidhat/llama-4-scout-17b-construction-codecite-v3

Llama 4 Scout 17B-16E — Construction Code-Citation v3 (AutoScientist, "win tier")

LoRA adapter on top of Llama 4 Scout 17B-16E-Instruct (109B total params, 17B active per forward pass, Mixture-of-Experts with 16 experts), trained by AutoScientist from Adaption Labs on the construction-code-corpus-v1 dataset (data-adapted variant, grade A). Predicts OIICS hazard codes (event, source, nature, body) and OSHA 29 CFR 1926 citations from construction-site incident narratives.

Built for the Adaption Labs AutoScientist Challenge, "All Other Domains" category. Credit to Adaptive Data by Adaption.

This is the primary submission model. See v2 (Llama 3.2 3B) and v1.1 (Qwen 2.5 1.5B) for smaller-tier comparisons on the same data.

Headline results

• 82% win rate for adapted vs base Llama 4 Scout 17B-16E on our OSHA test set.
• 72% Governance win rate — the adapted model also beat base on the broader Governance benchmark (categories orthogonal to construction). This is unusual: most fine-tunes trade generic capability for niche specialization. v3 kept both.
• Data quality: 6.0 → 9.1 (grade C → A, +51.7% relative) via AutoScientist's data adaption alone.
• Training end-to-end by AutoScientist — no hand-tuning of rank, alpha, LR schedule, or mixture ratios.

Inputs / Outputs

Input: free-text construction-site incident narrative. Output: strict JSON with hazards[] (4 OIICS codes + severity) and citations[] (verified OSHA 1926 standards).

{
  "hazards": [
    {
      "code_event":  {"id": "21",   "title": "Slip or fall without fall to lower level"},
      "code_source": {"id": "5510", "title": "Ice, snow"},
      "code_nature": {"id": "220",  "title": "Fractures"},
      "code_body":   {"id": "280",  "title": "Trunk"},
      "severity": "high"
    }
  ],
  "citations": [
    {"standard": "1926.501", "section_heading": "Duty to have fall protection", "verified": true}
  ]
}

Usage

Hosted (recommended, ~$0.20/M tokens):

from together import Together

client = Together()
response = client.chat.completions.create(
    model="rigidhat/llama-4-scout-17b-construction-codecite-v3",
    messages=[{"role": "user", "content": "<RAG-augmented prompt with narrative>"}],
    max_tokens=384,
    temperature=0.0,
)

Local (109B model requires substantial VRAM — recommend 4-bit BnB base):

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = "meta-llama/Llama-4-Scout-17B-16E-Instruct"
adapter = "rigidhat/llama-4-scout-17b-construction-codecite-v3"

tokenizer = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, adapter)

For the full RAG-augmented prompt template + citation verifier, see hf_space/app.py in the source repo.

Training recipe (from AutoScientist)

Field	Value
Base model	meta-llama/Llama-4-Scout-17B-16E-Instruct (trained on Together's 4-bit BnB variant)
Base params (total / active)	109B / 17B
Experts	16
Training method	SFT (LoRA)
LoRA rank	64
LoRA alpha	128
LoRA dropout	0.05
Target modules	attention (k_proj, o_proj, q_proj, v_proj) + shared_expert (gate_proj, up_proj, down_proj) + per-expert feed_forward (gate_proj, up_proj, down_proj)
Optimizer	AdamW
LR schedule	Cosine, warmup 5%, min_lr_ratio 0.1, num_cycles 0.5
Peak LR	1.5e-4
Weight decay	0.02
Grad clip	1.0
Epochs	5
Total steps	165
Batch size	max (device-fit)
Train on inputs	False (mask-prompt loss)
Eval checkpoints	5 (steps 33, 66, 99, 132, 165)
Train records	7,199 (post-adaption)
Train loss	1.03 → 0.58 (min 0.27)
Validation loss	0.75 → 0.68 (monotone decrease, no overfit)

Data

Trained on rigidhat/construction-code-corpus-v1 — 18,122 SFT records built from the OSHA Severe Injury Reports corpus (2015–2025), stratified 70/15/15 by NAICS subsector, with test split hash-pinned SHA-256 c9490ed3... and never opened during development.

AutoScientist data adaption applied: Reasoning Traces, Hallucination Mitigation, Prompt Rephrase, Prompt Deduplication. Post-adaption training set: 7,199 records at grade A (9.1/10).

Demo

Live Gradio Space (Together-hosted inference): rigidhat/construction-code-cite

Model lineage

Version	Base	Params	Recipe	Task win vs base	Governance win
v1.1	Qwen 2.5 1.5B	1.5B	Hand-tuned QLoRA, 400 steps	—	—
v2	Llama 3.2 3B	3B	AutoScientist, rank 32, 3 epochs	77%	46% (lost)
v3 (this)	Llama 4 Scout 17B-16E	109B / 17B active	AutoScientist, rank 64, 5 epochs	82%	72% (won)

The v3 story is the Governance flip: v2 traded broad capability for niche specialization (46/54 Governance). v3 gave up nothing — it beat base on both the OSHA task and the broader Governance category.

Source

Full pipeline (data pull, verifier, RAG index, eval harness): github.com/snakezilla/construction-code-llm

License

Llama 4 Community License applies to the base model. LoRA adapter weights released under MIT.

Citation

@misc{construction-codecite-v3-2026,
  title  = {Llama 4 Scout 17B-16E - Construction Code-Citation v3 (AutoScientist)},
  author = {Oversite Innovations},
  year   = {2026},
  note   = {Trained by AutoScientist for the Adaption Labs AutoScientist Challenge, All Other Domains category. 82% win rate on task, 72% on Governance.}
}