ViSceT5-ViOCRVQA

ViSceT5 is a Vietnamese scene-text visual question answering model: a ViT5 encoder-decoder that reads the image, the question and the OCR tokens of the image, and generates the answer. This checkpoint is the full model fine-tuned on ViOCRVQA.

Model overview

  • Backbone: VietAI/vit5-base (encoder-decoder), about 346M parameters in total (fp32, 1.38 GB).
  • QA-ViT: question-aware CLIP ViT-B/16 vision features (openai/clip-vit-base-patch16) with a gated adapter and a grounding loss.
  • OCR input: OCR text with box, detection, recognition and character features, fed as a block of the encoder input.
  • Consformer: self-attention with a constituent prior over OCR words, and a span loss.
  • DOVE: training on two views of the OCR block, the original and a "related" copy in which a few OCR words are edited.
  • OACT (OCR Adversarial Contrastive Training): a training strategy, not an inference module. Each OCR view gets a noise step along the sign of its answer-loss gradient (epsilon 0.05 of the token RMS), and a joint supervised-contrastive loss ties the four views (two clean, two perturbed; separate projection heads for clean and perturbed). It is switched on in this checkpoint (dove_adv=ocr_both) and unused at inference.

Results (ViOCRVQA)

Test: 18,601 questions, the full test split (ground truth provided separately; the public test JSON has placeholder answers). Main metric: EM / F1. Scored with training/benchmark_eval.py of the code repository (the benchmark protocol: EM, F1, BLEU, CIDEr, BERTScore with mBERT). Dev is the validation split at the selected checkpoint.

Metric Test Dev
Exact Match (EM) 61.68 61.73
F1 (token) 79.50 79.39
F1 (char) 87.14 -
BLEU-1 80.91 81.19
BLEU-4 73.54 73.65
BLEU-avg 77.16 -
CIDEr 6.1498 6.1934
BERTScore F1 93.31 -

Single training run (one seed); the checkpoint was selected on the dev set, never on test.

Usage

The architecture is custom (model_type: openvivqa), so it does not load with AutoModel. Use the code at commit 0907f9a, which is the code this checkpoint was trained with, and follow the notebook notebooks/twc_colab.ipynb (cell 6b, "predict from a Hugging Face repo"). In short:

git clone https://github.com/Kussssssss/ViSceT5 && cd ViSceT5 && git checkout 0907f9a
bash setup.sh
huggingface-cli download Kus669/ViSceT5-ViOCRVQA --local-dir ckpt
MOCK_TEST=false STAGE=predict PREDICT_SPLIT=both PREDICT_DATASET=ViOCRVQA \
  PREDICT_CKPT_DIR=ckpt HF_REPO= python3 run_pipeline.py

The script prepares the dataset and its OCR features, generates answers with beam search (4 beams, up to 56 new tokens) and writes a submission CSV.

Limitations

  • Trained and evaluated on ViOCRVQA only; no claim is made for other Vietnamese VQA data or other languages.
  • The answer depends on the OCR tokens: errors or missing text in the OCR output limit what the model can answer.
  • Results are from a single seed.

Citation

A paper describing ViSceT5 is in preparation; the citation will be added here.

Downloads last month
15
Safetensors
Model size
0.3B params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for Kus669/ViSceT5-ViOCRVQA

Base model

VietAI/vit5-base
Finetuned
(112)
this model

Collections including Kus669/ViSceT5-ViOCRVQA