rodriguescarson/polychart-shown-is-not-supported-lora

PolyChart — Shown Is Not Supported (LoRA)

A LoRA adapter for Llama-3.3-70B-Instruct that reads charts under a measure-then-claim contract: a claim is made only when the underlying values support it, and the distance between what a chart shows and what its data supports is reported as a graded quantity rather than a yes/no label.

Trained on Adaption's AutoScientist for the AutoScientist Challenge 2026.

Scores

	Value
Held-out category win rate (data-analysis-visualization)	85.9
Judge tally vs un-adapted base	74 wins / 5 losses / 17 ties
On-dataset win rate	84.6

Read the held-out figure. The on-dataset number is computed on the training distribution and the two are not interchangeable — elsewhere in this study a matched pair moved 87.2 → 81.0 on-dataset while moving 66.3 → 77.3 held-out, so optimising the displayed number can select the weaker model.

How this number was selected, and what it is worth

85.9 is the best of 7 identical runs. Same rows, same +2,000 augmentation dose, same base, same recipe. The other six scored 74.5, 78.5, 79.4, 80.2, 80.7 and 81.8.

That spread is not an accident of this adapter, it is a property of the platform's evaluation. Across 41 runs in 8 replicate groups we measured identical configurations differing by a median of 6.3 held-out points and by as much as 11.4. A single reported win rate is a draw from that band rather than a measurement of the dataset.

So this card reports the maximum and the sample it came from. Comparing it to another model's single reported number, point against point, would be unsound — the honest comparison is interval against interval. A model card whose subject is the gap between what is shown and what is supported should not open by hiding its own error bar.

What is novel here: severity as a continuous target

Released misleading-chart datasets, including Misviz (ACL 2026), ask a binary question: is this chart misleading? This model is trained on Tufte's Lie Factor as a continuous, signed target — −1.0 where an inverted axis reverses a real trend, 0.017 where an expanded axis flattens one, 4.69 where a truncated baseline invents one. A yes/no label collapses all three into the same bit.

The seed is failure-weighted. Scoring the base model cell-by-cell over question-type × distortion located the hard cells — continuous severity estimation, precise value extraction under a distorted axis, and mechanism naming — and the 2,129-row seed was weighted toward them (severity 27%), with already-solved cells removed.

Training

Base	Llama-3.3-70B-Instruct
Method	LoRA, r=64, alpha=128
Target modules	all linear
Epochs	3
Seed rows	2,129, failure-weighted
Augmentation	+2,000 synthetic domain rows

Every seed row passed an independent verifier sharing no code with the generator. It caught nine silent failures in our own pipeline before shipping, four of them inside its own scaffolding.

A retraction

An earlier version of this card claimed "curation beats scale," on the basis that expanding a seed from 1,882 to 21,882 rows dropped the held-out category from 76 to 72. That claim is withdrawn. It rested on one unpaired comparison, and 4 points sits well inside the 6.3-point replicate band measured above. Repeated properly — paired within seed across 13 seeds and 176 jobs — augmentation improved the held-out score on 13 of 13 seeds, mean +14, two-sided sign test p = 0.0002, and the platform's advertised "+16 beyond 20,000 datapoints" reproduced at +17.4.

It is recorded rather than deleted because it is the same error the rest of this card exists to warn about: reading a single noisy draw as a result.

Intended use and limits

Chart-interpretation QA and misleading-encoding detection. Not a general-purpose assistant.

• One task family, one platform, one base model.
• The evaluation domain is assigned by the platform, not chosen. Across six purpose-built datasets, five were scored against general and none against its own topic. This adapter is one of the few scored against data-analysis-visualization.
• Held-out evaluations fire on a fraction of jobs and cannot be triggered on demand.
• Inherits the biases and failure modes of Llama-3.3-70B-Instruct.

Use

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = "meta-llama/Llama-3.3-70B-Instruct"
model = AutoModelForCausalLM.from_pretrained(base, device_map="auto")
model = PeftModel.from_pretrained(model, "rodriguescarson/polychart-shown-is-not-supported-lora")
tok = AutoTokenizer.from_pretrained(base)

Links

• Client + platform findings: https://github.com/rodriguescarson/adaption-autoscientist-api
• Live Lie Factor demo: https://lie-factor.carsonrodrigues.com
• Datasets: https://huggingface.co/rodriguescarson

License

Llama 3.3 Community License, inherited from the base model.