ViSceT5-OpenViVQA

ViSceT5 is a Vietnamese scene-text visual question answering model: a ViT5 encoder-decoder that reads the image, the question and the OCR tokens of the image, and generates the answer. This checkpoint is the full model fine-tuned on OpenViVQA.

Model overview

  • Backbone: VietAI/vit5-base (encoder-decoder), about 346M parameters in total (fp32, 1.38 GB).
  • QA-ViT: question-aware CLIP ViT-B/16 vision features (openai/clip-vit-base-patch16) with a gated adapter and a grounding loss.
  • OCR input: OCR text with box, detection, recognition and character features, fed as a block of the encoder input.
  • Consformer: self-attention with a constituent prior over OCR words, and a span loss.
  • DOVE: training on two views of the OCR block, the original and a "related" copy in which a few OCR words are edited.
  • OACT (OCR Adversarial Contrastive Training): a training strategy, not an inference module. Each OCR view gets a noise step along the sign of its answer-loss gradient (epsilon 0.05 of the token RMS), and a joint supervised-contrastive loss ties the four views (two clean, two perturbed; separate projection heads for clean and perturbed). It is switched on in this checkpoint (dove_adv=ocr_both) and unused at inference.

Results (OpenViVQA)

Test: 3,536 questions, the subset of the test split with public ground truth. Main metric: CIDEr (answers are free-form). Scored with training/benchmark_eval.py of the code repository (the benchmark protocol: EM, F1, BLEU, CIDEr, BERTScore with mBERT). Dev is the validation split at the selected checkpoint.

Metric Test Dev
Exact Match (EM) 13.21 13.71
F1 (token) 58.76 58.59
F1 (char) 69.36 -
BLEU-1 55.52 55.23
BLEU-4 41.23 41.09
BLEU-avg 47.85 -
CIDEr 3.7202 3.7568
BERTScore F1 85.85 -

Single training run (one seed); the checkpoint was selected on the dev set, never on test.

Usage

The architecture is custom (model_type: openvivqa), so it does not load with AutoModel. Use the code at commit 0907f9a, which is the code this checkpoint was trained with, and follow the notebook notebooks/twc_colab.ipynb (cell 6b, "predict from a Hugging Face repo"). In short:

git clone https://github.com/Kussssssss/ViSceT5 && cd ViSceT5 && git checkout 0907f9a
bash setup.sh
huggingface-cli download Kus669/ViSceT5-OpenViVQA --local-dir ckpt
MOCK_TEST=false STAGE=predict PREDICT_SPLIT=both PREDICT_DATASET=OpenViVQA \
  PREDICT_CKPT_DIR=ckpt HF_REPO= python3 run_pipeline.py

The script prepares the dataset and its OCR features, generates answers with beam search (4 beams, up to 56 new tokens) and writes a submission CSV.

Limitations

  • Trained and evaluated on OpenViVQA only; no claim is made for other Vietnamese VQA data or other languages.
  • The answer depends on the OCR tokens: errors or missing text in the OCR output limit what the model can answer.
  • Results are from a single seed.

Citation

A paper describing ViSceT5 is in preparation; the citation will be added here.

Downloads last month
53
Safetensors
Model size
0.3B params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for Kus669/ViSceT5-OpenViVQA

Base model

VietAI/vit5-base
Finetuned
(112)
this model

Collections including Kus669/ViSceT5-OpenViVQA