How to use from the
Use from the
Transformers library
# Use a pipeline as a high-level helper
# Warning: Pipeline type "visual-question-answering" is no longer supported in transformers v5.
# You must load the model directly (see below) or downgrade to v4.x with:
# pip install "transformers<5.0.0"
from transformers import pipeline

pipe = pipeline("visual-question-answering", model="Kus669/ViSceT5-ViTextVQA")
# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("Kus669/ViSceT5-ViTextVQA", device_map="auto")
Quick Links

ViSceT5-ViTextVQA

ViSceT5 is a Vietnamese scene-text visual question answering model: a ViT5 encoder-decoder that reads the image, the question and the OCR tokens of the image, and generates the answer. This checkpoint is the full model fine-tuned on ViTextVQA.

Model overview

  • Backbone: VietAI/vit5-base (encoder-decoder), about 346M parameters in total (fp32, 1.38 GB).
  • QA-ViT: question-aware CLIP ViT-B/16 vision features (openai/clip-vit-base-patch16) with a gated adapter and a grounding loss.
  • OCR input: OCR text with box, detection, recognition and character features, fed as a block of the encoder input.
  • Consformer: self-attention with a constituent prior over OCR words, and a span loss.
  • DOVE: training on two views of the OCR block, the original and a "related" copy in which a few OCR words are edited.
  • OACT (OCR Adversarial Contrastive Training): a training strategy, not an inference module. Each OCR view gets a noise step along the sign of its answer-loss gradient (epsilon 0.05 of the token RMS), and a joint supervised-contrastive loss ties the four views (two clean, two perturbed; separate projection heads for clean and perturbed). It is switched on in this checkpoint (dove_adv=ocr_both) and unused at inference.

Results (ViTextVQA)

Test: 10,028 questions, the full public test split. Main metric: EM / F1. Scored with training/benchmark_eval.py of the code repository (the benchmark protocol: EM, F1, BLEU, CIDEr, BERTScore with mBERT). Dev is the validation split at the selected checkpoint.

Metric Test Dev
Exact Match (EM) 36.01 34.51
F1 (token) 58.93 59.46
F1 (char) 72.27 -
BLEU-1 58.88 58.28
BLEU-4 45.12 44.70
BLEU-avg 51.65 -
CIDEr 3.3181 3.3852
BERTScore F1 87.25 -

Ablation on ViTextVQA (test)

Each row is a separate training run (single seed); row 4 uses batch size 32, the others 16.

Configuration EM F1
Baseline (ViT5 + question) 29.94 54.91
+ Consformer 32.55 56.40
+ QA-ViT 32.88 56.77
+ DOVE (batch 32) 34.60 58.26
+ QA-ViT + Consformer + DOVE 34.85 58.20
+ OACT (this model) 36.01 58.93

Differences below about 0.6 EM are within the run-to-run noise we observed, so read the gain of each module as indicative, not as a significance test.

Single training run (one seed); the checkpoint was selected on the dev set, never on test.

Usage

The architecture is custom (model_type: openvivqa), so it does not load with AutoModel. Use the code at commit d350d49, which is the code this checkpoint was trained with, and follow the notebook notebooks/twc_colab.ipynb (cell 6b, "predict from a Hugging Face repo"). In short:

git clone https://github.com/Kussssssss/ViSceT5 && cd ViSceT5 && git checkout d350d49
bash setup.sh
huggingface-cli download Kus669/ViSceT5-ViTextVQA --local-dir ckpt
MOCK_TEST=false STAGE=predict PREDICT_SPLIT=both PREDICT_DATASET=ViTextVQA \
  PREDICT_CKPT_DIR=ckpt HF_REPO= python3 run_pipeline.py

The script prepares the dataset and its OCR features, generates answers with beam search (4 beams, up to 56 new tokens) and writes a submission CSV.

Limitations

  • Trained and evaluated on ViTextVQA only; no claim is made for other Vietnamese VQA data or other languages.
  • The answer depends on the OCR tokens: errors or missing text in the OCR output limit what the model can answer.
  • Results are from a single seed.

Citation

A paper describing ViSceT5 is in preparation; the citation will be added here.

Downloads last month
52
Safetensors
Model size
0.3B params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for Kus669/ViSceT5-ViTextVQA

Base model

VietAI/vit5-base
Finetuned
(112)
this model

Collections including Kus669/ViSceT5-ViTextVQA