Instructions to use Kus669/ViSceT5-ViTextVQA with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Kus669/ViSceT5-ViTextVQA with Transformers:
# Use a pipeline as a high-level helper # Warning: Pipeline type "visual-question-answering" is no longer supported in transformers v5. # You must load the model directly (see below) or downgrade to v4.x with: # pip install "transformers<5.0.0" from transformers import pipeline pipe = pipeline("visual-question-answering", model="Kus669/ViSceT5-ViTextVQA")# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("Kus669/ViSceT5-ViTextVQA", device_map="auto") - Notebooks
- Google Colab
- Kaggle
ViSceT5-ViTextVQA
ViSceT5 is a Vietnamese scene-text visual question answering model: a ViT5 encoder-decoder that reads the image, the question and the OCR tokens of the image, and generates the answer. This checkpoint is the full model fine-tuned on ViTextVQA.
Model overview
- Backbone: VietAI/vit5-base (encoder-decoder), about 346M parameters in total (fp32, 1.38 GB).
- QA-ViT: question-aware CLIP ViT-B/16 vision features (openai/clip-vit-base-patch16) with a gated adapter and a grounding loss.
- OCR input: OCR text with box, detection, recognition and character features, fed as a block of the encoder input.
- Consformer: self-attention with a constituent prior over OCR words, and a span loss.
- DOVE: training on two views of the OCR block, the original and a "related" copy in which a few OCR words are edited.
- OACT (OCR Adversarial Contrastive Training): a training strategy, not an inference module. Each OCR view gets a noise step
along the sign of its answer-loss gradient (epsilon 0.05 of the token RMS), and a joint supervised-contrastive loss ties the four views
(two clean, two perturbed; separate projection heads for clean and perturbed). It is switched on in this checkpoint
(
dove_adv=ocr_both) and unused at inference.
Results (ViTextVQA)
Test: 10,028 questions, the full public test split. Main metric: EM / F1. Scored with training/benchmark_eval.py of the code repository
(the benchmark protocol: EM, F1, BLEU, CIDEr, BERTScore with mBERT). Dev is the validation split at the selected checkpoint.
| Metric | Test | Dev |
|---|---|---|
| Exact Match (EM) | 36.01 | 34.51 |
| F1 (token) | 58.93 | 59.46 |
| F1 (char) | 72.27 | - |
| BLEU-1 | 58.88 | 58.28 |
| BLEU-4 | 45.12 | 44.70 |
| BLEU-avg | 51.65 | - |
| CIDEr | 3.3181 | 3.3852 |
| BERTScore F1 | 87.25 | - |
Ablation on ViTextVQA (test)
Each row is a separate training run (single seed); row 4 uses batch size 32, the others 16.
| Configuration | EM | F1 |
|---|---|---|
| Baseline (ViT5 + question) | 29.94 | 54.91 |
| + Consformer | 32.55 | 56.40 |
| + QA-ViT | 32.88 | 56.77 |
| + DOVE (batch 32) | 34.60 | 58.26 |
| + QA-ViT + Consformer + DOVE | 34.85 | 58.20 |
| + OACT (this model) | 36.01 | 58.93 |
Differences below about 0.6 EM are within the run-to-run noise we observed, so read the gain of each module as indicative, not as a significance test.
Single training run (one seed); the checkpoint was selected on the dev set, never on test.
Usage
The architecture is custom (model_type: openvivqa), so it does not load with AutoModel. Use the code at commit
d350d49, which is the code this checkpoint was trained with, and follow the notebook
notebooks/twc_colab.ipynb (cell 6b, "predict from a Hugging Face repo"). In short:
git clone https://github.com/Kussssssss/ViSceT5 && cd ViSceT5 && git checkout d350d49
bash setup.sh
huggingface-cli download Kus669/ViSceT5-ViTextVQA --local-dir ckpt
MOCK_TEST=false STAGE=predict PREDICT_SPLIT=both PREDICT_DATASET=ViTextVQA \
PREDICT_CKPT_DIR=ckpt HF_REPO= python3 run_pipeline.py
The script prepares the dataset and its OCR features, generates answers with beam search (4 beams, up to 56 new tokens) and writes a submission CSV.
Limitations
- Trained and evaluated on ViTextVQA only; no claim is made for other Vietnamese VQA data or other languages.
- The answer depends on the OCR tokens: errors or missing text in the OCR output limit what the model can answer.
- Results are from a single seed.
Citation
A paper describing ViSceT5 is in preparation; the citation will be added here.
- Downloads last month
- 50
Model tree for Kus669/ViSceT5-ViTextVQA
Base model
VietAI/vit5-base