README.md

---
license: other
license_name: llama4
license_link: https://github.com/meta-llama/llama-models/blob/main/models/llama4/LICENSE
language:
  - en
base_model: meta-llama/Llama-4-Scout-17B-16E-Instruct
tags:
  - lora
  - text-to-visualization
  - nl2vis
  - nvbench
  - data-visualization
  - closed-label-classification
  - adaption
  - sft
library_name: peft
pipeline_tag: text-generation
datasets:
  - Fernandosr85/adaption-nl-to-viz-spec-pairs
---

![banner](https://proteus-prod-public.s3.us-east-1.amazonaws.com/temp/f6c5e1d2-b810-4926-93a5-b3758c769be7.png)

# NL2VIS Chart Spec Adapter

LoRA adapter for converting natural-language chart requests into visualization specifications, fine-tuned on **Llama-4-Scout-17B-16E-Instruct** (109B parameters across 16 experts, 17B active per token) via [Adaption's](https://adaptionlabs.ai) AutoScientist platform.

Given a database schema and a request in plain English, the adapter emits one line with five fields:

```
CHART_TYPE || X_COLUMN || Y_COLUMN || GROUP_COLUMN || AGGREGATE
```

---

## Task

**Closed-vocabulary specification, not open generation.** Every field comes from a vocabulary external to the model:

| Field | Vocabulary | Source |
|---|---|---|
| `CHART_TYPE` | BAR, LINE, SCATTER, PIE | nvBench |
| `X_COLUMN` | columns of the schema in the prompt | Spider `tables.json` |
| `Y_COLUMN` | columns, or `*` for a plain row count | Spider `tables.json` |
| `GROUP_COLUMN` | columns, or `NONE` | Spider `tables.json` |
| `AGGREGATE` | NONE, COUNT, SUM, AVG, MIN, MAX | SQL grammar |

Because the column vocabulary is the schema supplied in the prompt, a predicted column outside it is a **checkable hallucination** — a set lookup, not a judgement call. That splits column errors in two:

```
COLUMN_HALLUCINATION   the column does not exist in the schema
COLUMN_SCOPE_ERROR     the column exists but sits on the wrong channel
```

Measured on the holdout, two frontier models produced **0.0% hallucination**: the output stays inside the schema, and every column error is a scope error.

---

## Evaluation results

### Training win rates (Adaption internal)

| Model | Win Rate (in-domain) |
|---|---|
| Base (`Llama-4-Scout-17B-16E-Instruct`) | 2% |
| **Adapted (`adaption_nl_to_viz_spec_pairs`)** | **98%** |

A **+96 point** improvement.

**Cross-domain, without forgetting.** On the Data-analysis-visualization reference category the win rate rises **45 → 56**. The adapter improved outside its own dataset rather than trading general capability for in-domain accuracy — worth noting because task-specific adapters usually pay that price.

### How to read a base score of 2

Two frontier models were measured on this exact task at **45.7%** and **34.7%** full exact-match. A base score of 2 therefore does not mean the model cannot reason about visualization. It means it does not emit a parseable five-field spec: win rate compares responses, and prose describing a chart loses to a specification every time.

A large share of those 96 points is **format adherence** rather than visualization competence. That is precisely what makes the adapter usable in a rendering pipeline, but the honest headline is *"the base model does not emit a parseable spec; the adapter does, and gets it right"* — not *"the base model cannot do visualization"*.

### Zero-shot baselines on the published holdout

300 rows, text-only, two scoring regimes. Strict counts `COUNT(<column>)` against `COUNT(*)` as an error; equivalence-aware does not, because they render the identical chart.

| Method | Strict full EM | Equivalence-aware |
|---|---|---|
| Majority (`BAR`/`NONE`/`COUNT`) | 0.0% | — |
| Lexical (best name-overlap column) | 9.3% | — |
| Frontier model A | 45.7% | 67.0% |
| Frontier model B | 34.7% | 68.0% |
| **This adapter** | `[not measured]` | `[not measured]` |

The lexical baseline is the control that matters: it picks the schema column whose name best overlaps the request words, and always answers `BAR` / `NONE`. A model that cannot clearly beat it is doing string matching rather than visualization grounding. The margin was **+36.3 points**.

---

## Two measurement artifacts found in nvBench

**1. `COUNT(<column>)` and `COUNT(*)` are the same chart.**

38.7% of the corpus has `AGGREGATE=COUNT` with a named column as gold `y_column`. And:

```
100% − 38.7%                          = 61.3%
y_column accuracy, model A (measured) = 61.3%
y_column accuracy, model B (measured) = 61.7%
```

Not a coincidence. Both models got `y_column` right everywhere **except** where gold writes `COUNT(paperID)` and they answer `COUNT(*)`. Scoring leniently under COUNT moves `y_column` from 61.3% to 96.3%, and full exact-match from 34.7% to 67.0%.

**2. `vis_obj.classify` is not the grouping column.**

It holds the distinct *values* of that column — `['F','M']` for `GROUP BY Sex, Rank`. The name must be parsed from the SQL `GROUP BY` clause, taking the term that is not the x column. A parser reading `classify` as a column name produces silently wrong gold on the 7.7% of records that are grouped charts.

Related: `AGGREGATE` parsed from `y_name` disagrees with the SQL on 3.4% of records. The SQL is the specification; `y_name` is a display string.

---

## Two fields cannot carry a claim

| Field | Majority value | Share |
|---|---|---|
| `CHART_TYPE` | `bar` | **81.2%** |
| `GROUP_COLUMN` | `NONE` | **92.3%** |

Always answering `bar` scores 81.2%; always answering `NONE` scores 92.3%. A frontier model reached 89.7% on `group_column` — **0.7 points above the majority baseline**, i.e. no signal at all.

The headroom sits in `AGGREGATE` (84.3% against a 54.2% majority) and `x_column` (92.7%), and the training mixture is stratified accordingly.

---

## The hypothesis that did not survive

This work was built to test whether the failure mode is **column attribution**: when a schema offers confusable columns (`sales_amount`/`sales_count`, `region`/`sub_region`), does the model attach the right one to the right channel?

A confusable slice covering 50.3% of records was defined in advance, with a 10-point threshold. The measured gap was **3.2 points**. The models do degrade on confusable schemas, but not much, and the headroom is not concentrated where the hypothesis predicted.

It is stated here because a diagnostic that reports only its confirmations is not a diagnostic.

---

## Model details

| Field | Value |
|---|---|
| Base model | `meta-llama/Llama-4-Scout-17B-16E-Instruct` (109B total, 17B active per token, 16 experts) |
| Trained model name | `adaption_nl_to_viz_spec_pairs` |
| Training method | Supervised Fine-Tuning (SFT) + LoRA |
| LoRA rank (r) | 64 |
| LoRA alpha | 128 |
| LoRA dropout | 0 |
| Trainable modules | attention (`q/k/v/o_proj`) **plus** expert FFN (`feed_forward.*`) and shared-expert FFN (`shared_expert.*`) |
| Epochs | 5 |
| Learning rate | 1e-4 (cosine, warmup 0.05) |
| Weight decay | 0.03 |
| Max grad norm | 1.0 |
| Train on inputs | false |
| Data format | chat |
| Adapter size | 829 MB |

**On the module selection.** Unlike an attention-only adapter, this configuration reaches the mixture-of-experts feed-forward layers — both the routed experts and the shared expert. On a 109B-parameter MoE base that is where most of the parametric knowledge sits, and it accounts for the 829 MB footprint against the ~200 MB of a comparable attention-only LoRA.

**On the model name.** The `17B` in `Llama-4-Scout-17B-16E` is the *active* parameter count per token, not the model size: the base carries 109B parameters across 16 experts and routes a subset per token. Inference needs VRAM for the full footprint — sparse activation reduces compute, not memory.

**Inference note.** `data_format` was `chat` and `train_on_inputs` was `false`. Wrap the request as a user turn via the tokenizer's chat template; raw text will not match the training format.

**Archive note.** The distributed archive may be **zstd-compressed despite a `.tgz` extension**. If `tar -xzf` fails, use `tar --use-compress-program=unzstd -xf`.

---

## Usage

```python
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = "meta-llama/Llama-4-Scout-17B-16E-Instruct"
tokenizer = AutoTokenizer.from_pretrained(base)
base_model = AutoModelForCausalLM.from_pretrained(
    base, torch_dtype=torch.float16, device_map="auto",
)
model = PeftModel.from_pretrained(
    base_model, "Fernandosr85/nl2vis-chart-spec-adapter",
)

# The field definitions are part of the trained prompt. GROUP_COLUMN in
# particular must be spelled out: an earlier prompt said only "a schema column
# or NONE", and the model answered with the x column again, scoring 72.0%
# against an 89.0% majority baseline -- worse than always answering NONE.
prompt = """You are converting a natural-language request into a visualization specification.

DATABASE SCHEMA:
- Region_name [text]
- Country [text]
- Population [number]
- Area_km2 [number]

REQUEST:
Show the average population by country as a bar chart.

Answer with exactly one line in this format, nothing else:
CHART_TYPE || X_COLUMN || Y_COLUMN || GROUP_COLUMN || AGGREGATE

CHART_TYPE: one of BAR, LINE, SCATTER, PIE.

X_COLUMN: the schema column plotted on the horizontal axis, copied verbatim.

Y_COLUMN: the schema column the aggregate is computed over, copied verbatim.
Use * ONLY when the request asks for a plain count of rows with no column named.
If the request counts or sums a specific column, name that column instead.

GROUP_COLUMN: the extra series dimension that splits the data into multiple
series, as in a grouped or stacked chart. It MUST be different from X_COLUMN.
Answer NONE for an ordinary single-series chart. Most charts are single-series,
so NONE is the common answer. Do not repeat X_COLUMN here.

AGGREGATE: one of NONE, COUNT, SUM, AVG, MIN, MAX."""

messages = [{"role": "user", "content": prompt}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt",
).to(model.device)
outputs = model.generate(inputs, max_new_tokens=32, do_sample=False)
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))

# Expected:
# BAR || Country || Population || NONE || AVG
#
# Validate before rendering: every column in the output should exist in the
# schema you supplied. That is a set lookup, and it is worth running.
```

`do_sample=False` on purpose — specification is deterministic, and you want the most likely label rather than a sample.

---

## Training dataset

**[Fernandosr85/adaption-nl-to-viz-spec-pairs](https://huggingface.co/datasets/Fernandosr85/adaption-nl-to-viz-spec-pairs)**

13,635 prompt/completion pairs derived from **nvBench** (7,247 VIS records, 25,762 NL queries, 152 databases), with schemas resolved against **Spider** — all 152 matched, 0 missing. nvBench labels were validated by 23 experts and 300+ crowd workers.

| Step | Detail |
|---|---|
| Label source | nvBench, external to this project |
| Gold parsing | `AGGREGATE` from the SQL, not `y_name` (3.4% disagreement) |
| Group column | From the SQL `GROUP BY`, the term that is not x |
| Split | By VIS key; no eval key in the training set (asserted in code) |
| NL per record | 2 of the up-to-9 paraphrases nvBench ships |
| Stratification | By `AGGREGATE`, the field with measurable headroom |
| Rare-class boost | `MIN` and `MAX` 3× from their 1.3% and 1.5% natural share |
| Completions | Kept verbatim, **not** platform-regenerated |

**On the COUNT normalisation.** Under COUNT the y column carries no visual information, so all COUNT cases were normalised to `*`. Training on the gold as-is would teach the adapter to guess which arbitrary column the annotator happened to write.

**On what the mixture bets on.** `CHART_TYPE` and `Y_COLUMN` are near-solved by frontier models (99.0% and 96.3%) and are included only to keep the output format intact. `GROUP_COLUMN` has no signal. The bet is `AGGREGATE`.

---

## Known limitations

- **Holdout accuracy is unmeasured.** The win rate above is Adaption's internal metric; the adapter has not been scored against the published 300-row holdout.
- **Structural only.** The spec says which chart type, axes, grouping and aggregate. It says nothing about whether that is a *good* way to present the data. Encoding quality is a separate question this does not measure.
- **Columns must be in the prompt.** The adapter copies from the schema it is given; it has no knowledge of any particular database.
- **Academic schemas.** nvBench is synthesised from Spider — single-database queries over 152 academic schemas, not production warehouses.

---

## Ethics

A visualization spec determines what a reader sees. A wrong aggregate turns a sum into an average; a wrong grouping column merges series that should be separate. Neither error announces itself in the rendered chart.

This adapter is a drafting aid for a pipeline a person reviews, not an autonomous charting agent. The five-field output is deliberately narrow so the spec can be validated against the schema before anything is rendered — column membership is a set lookup, and it should be run.

---

## Repositories

| Platform | Link |
|---|---|
| HuggingFace Model | [Fernandosr85/nl2vis-chart-spec-adapter](https://huggingface.co/Fernandosr85/nl2vis-chart-spec-adapter) |
| HuggingFace Dataset | [Fernandosr85/adaption-nl-to-viz-spec-pairs](https://huggingface.co/datasets/Fernandosr85/adaption-nl-to-viz-spec-pairs) |
| Kaggle Model | [nl2vis-chart-spec-adapter](https://www.kaggle.com/models/fernandosr85/nl2vis-chart-spec-adapter) |
| Kaggle Diagnostic | [nl2vis-zero-shot-headroom](https://www.kaggle.com/datasets/fernandosr85/nl2vis-zero-shot-headroom) |

---

## Credits

- **Fine-tuning platform:** [Adaption](https://adaptionlabs.ai) — AutoScientist & Adaptive Data
- **Challenge:** [AutoScientist Challenge 2026](https://adaptionlabs.ai/blog/autoscientist-challenge) — Data Visualization category
- **Source benchmark:** [nvBench](https://github.com/TsinghuaDatabaseGroup/nvBench) — Luo et al., *Synthesizing Natural Language to Visualization Benchmarks from NL2SQL Benchmarks*, SIGMOD 2021
- **Schemas:** [Spider](https://yale-lily.github.io/spider) — Yu et al., EMNLP 2018
- **Related evaluation:** [VisEval](https://github.com/microsoft/VisEval) — IEEE TVCG 2024
- **Author:** Fernando Rodrigues · [Kaggle: fernandosr85](https://www.kaggle.com/fernandosr85) · [HuggingFace: Fernandosr85](https://huggingface.co/Fernandosr85)

---

## Disclaimer

Experimental research artifact submitted to the AutoScientist Challenge 2026 (Data Visualization category). Holdout accuracy is unmeasured. Outputs are structural specifications and require review before rendering or downstream use.

Licensed under the Llama 4 Community License Agreement, inherited from the base model.