CivicInsight: Accessible Civic Data Descriptions

CivicInsight is a fine-tuned vision-language model that generates ARIA-ready descriptions of civic data dashboards (charts, maps, and visualizations) to improve accessibility for screen reader users. Built on Gemma 4 E4B with LoRA adapters on a curated dataset of 61 civic-data visualizations.

The model is designed to be paired with a deterministic grounding pipeline that checks every numeric value in the model's prose against a provided source CSV. Values that don't match any cell within tolerance are flagged as unmatched; values matching multiple cells without enough context to disambiguate are flagged as ambiguous; the rest are reported as confirmed. The pipeline does not address non-numeric claims (color, selection, layout, named entities) and can mis-confirm fabrications that coincidentally match a CSV cell with sufficient surrounding context. See the CivicInsight repository for the full grounding pipeline implementation.

Intended use

Primary: Generate alt-text and ARIA descriptions for civic data dashboards. The model extracts structured information (chart type, axes, values, selections, tooltips) and produces screen-reader-compatible descriptions in a consistent format prefixed with [civicinsight-v1].

Recommended deployment: Pair with the grounding pipeline in the project repository. Inference alone is appropriate for exploratory use; use cases that depend on numeric accuracy should always include the grounding step.

Out of scope:

General image captioning (use a general VLM)
Privacy-sensitive content (this model has not been adversarially tested for redaction)
Real-time captioning of dynamic content
Generation of charts (this is a description model, not a generation model)
Use without verification when source data is available, given the documented fabrication patterns under "Limitations" below

Architecture

Base model: unsloth/gemma-4-e4b-it, loaded in 4-bit at runtime via bitsandbytes
Adapter: LoRA, r=16, alpha=32, dropout=0
Trainable parameters: 39,403,520 (0.49% of 7,980,504,352 base parameters)
Vision tower: unfrozen during fine-tuning
Target modules: all-linear (attention Q/K/V/O and MLP gate/up/down across vision and language layers)
Training framework: Unsloth + TRL SFTTrainer + UnslothVisionDataCollator
Hardware: A100 80GB on Modal

Training data

61 civic-data visualizations:

50 base examples spanning multiple chart types: line charts, bar charts (including stacked), scatter plots, choropleth maps, box plots, gauges, tables, and pie charts
11 choropleth augmentations added to address documented choropleth perception failures: 10 categorical choropleths from French municipal election data, plus dual-encoded choropleths combining color and circle size

Source data primarily from French civic datasets (data.gouv.fr, licensed under Licence Ouverte 2.0) and international benchmark visualizations. Annotations follow a consistent rule sheet: marker prefix, chart-type slot opener, field-by-field structured description, no fabricated values, no hedging language when values are clearly visible.

The dataset is intentionally small. The architectural claim is that fine-tuning paired with deterministic verification together address civic data accessibility better than scale alone, and that grounding against source CSVs is more reliable than training the model to fabricate less often.

Training configuration: learning rate 2e-4, per-device batch 1 with gradient accumulation 4 (effective batch 4), 5 epochs, max sequence length 2048. Total of 80 update steps. Final adapter saved at checkpoint-80.

Model versions

main (default): canonical v1 release, exp4c-sft choropleth-augmented model.
v1.0 (tag): immutable pin to the v1 release commit. Use this revision in production code that needs reproducibility.

For most users, from_pretrained(revision="main") is correct. Pin to revision="v1.0" for code that should not drift if main moves forward in future releases.

Limitations

The model has documented failure modes from a held-out audit on diverse civic data visualizations. Inference with the grounding pipeline (CSV-backed) is recommended wherever possible, not the model alone.

Documented patterns:

Fabricated tooltips with externally-sourced facts. The model may pull values from its pretraining (e.g., for known countries, companies) rather than from the image. This is the highest accessibility risk pattern. When source data is provided and the fabricated value differs from any CSV cell, the grounding pipeline flags it as unmatched.
Confident invention. The model fabricates specific values (axis tick steps, percentages, counts) when uncertain. When source data is provided, the grounding pipeline flags these as unmatched if they're not in the CSV, or ambiguous if they coincidentally match without distinguishing context.
Positional binding errors on stacked bars. On multi-segment charts, the model may bind values to the wrong row or segment.
Color self-contradiction. The model occasionally references one color in one part of the description and a different color elsewhere in the same output.
False selection. On UI screenshots with multiple visible elements, the model may claim a selection state that does not exist.
Proper noun substitution. The model may swap obscure proper nouns for famous regional alternatives (e.g., "Sartène" rendered as "Ajaccio") while keeping surrounding numeric data accurate. High civic-data accessibility risk because famous-name substitutions read as plausible to readers unfamiliar with the data.
Structural fabrication. The model may invent a clean schema (e.g., "one tooltip per category") and force real data into it, even when the data does not support that schema.

These patterns are characteristic of small-data fine-tuning rather than civic data specifically; they apply broadly to vision-language models trained at this scale. The grounding pipeline addresses numeric mismatches against a provided CSV; non-numeric patterns (color, selection, layout, named entities) and coincidental matches with sufficient context overlap still require human review.

Grounding pipeline behavior

Two further caveats users should know about:

Conservative matcher. The grounding pipeline uses a context-window heuristic for entity-to-value disambiguation. When the model's prose places the entity label and the numeric value far apart in a sentence (e.g., "Ajaccio is marked in dark blue at the top of the 0-47 scale"), the matcher may report unverified even when the mapping is correct. This bias toward false negatives over false positives is intentional: an unverified flag prompts the user to check source data, while a false confirmation could mislead them.
Source CSV is not validated against the image. The pipeline does not check that the provided CSV semantically corresponds to the visualization. Users should provide the CSV that produced the chart. An unrelated CSV (e.g., school attendance data alongside a browser-share chart) will parse fine but produce "all unverified" output — accurate in the sense that no values match, but not actionable. A confidence score near 0% with most values unmatched is the signal that the CSV may be wrong for the image.

A DPO post-training stage was attempted to address several of these patterns but regressed model quality on held-out data. See the project repository for details. DPO is deferred to a v2 release with revised methodology.

Inference

The model is designed for use behind the grounding pipeline. Direct inference example:

from unsloth import FastVisionModel
from PIL import Image

model, tokenizer = FastVisionModel.from_pretrained(
    "shahfazal/civicinsight-gemma4-e4b-it",
    revision="v1.0",
    load_in_4bit=True,
)
FastVisionModel.for_inference(model)

image = Image.open("dashboard.png")
prompt = "Generate an aria-label for this data visualization image."

message = [{
    "role": "user",
    "content": [
        {"type": "image", "image": image},
        {"type": "text", "text": prompt},
    ],
}]
inputs = tokenizer.apply_chat_template(
    message,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=600)
description = tokenizer.decode(outputs[0], skip_special_tokens=True)

Recommended pattern: use the grounding pipeline wrapper in the project repository, which handles inference plus verification.

Reproducibility

Working stack as of May 2026:

unsloth         git+https://github.com/unslothai/unsloth.git@4f9c8321...
trl             0.24.0
torch           2.11.0+cu129  (matching torchvision from cu129 index)
transformers    5.5.0
peft            0.19.1
datasets        4.3.0
unsloth-zoo     2026.4.9
bitsandbytes    0.49.2
Python          3.12

Training notebook: notebooks/modal/07c-experiment-4c-sft.ipynb in the project repository.

Acknowledgements

@datta0 and the Unsloth team for fast turnaround on vision DPO bug fixes (issue #5196, PR #5199) which unblocked DPO experimentation
data.gouv.fr for source civic data under Licence Ouverte 2.0
The Kaggle Gemma 4 Good Hackathon team for the competition framing

License

The LoRA adapter weights in this repository are released under the MIT License.

The base Gemma 4 E4B model is subject to Google's Gemma terms of use, which apply independently to the base model regardless of how the adapter is licensed.

Citation

@misc{civicinsight2026,
  author = {Mohammed, Shahfazal},
  title = {CivicInsight: Accessible Civic Data Descriptions via Vision-Language Fine-Tuning and Deterministic Grounding},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {https://huggingface.co/shahfazal/civicinsight-gemma4-e4b-it},
}

Downloads last month: 3