Instructions to use Kus669/ViSceT5-ViOCRVQA with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Kus669/ViSceT5-ViOCRVQA with Transformers:
# Use a pipeline as a high-level helper # Warning: Pipeline type "visual-question-answering" is no longer supported in transformers v5. # You must load the model directly (see below) or downgrade to v4.x with: # pip install "transformers<5.0.0" from transformers import pipeline pipe = pipeline("visual-question-answering", model="Kus669/ViSceT5-ViOCRVQA")# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("Kus669/ViSceT5-ViOCRVQA", device_map="auto") - Notebooks
- Google Colab
- Kaggle
ViSceT5-ViOCRVQA
ViSceT5 is a Vietnamese scene-text visual question answering model: a ViT5 encoder-decoder that reads the image, the question and the OCR tokens of the image, and generates the answer. This checkpoint is the full model fine-tuned on ViOCRVQA.
Model overview
- Backbone: VietAI/vit5-base (encoder-decoder), about 346M parameters in total (fp32, 1.38 GB).
- QA-ViT: question-aware CLIP ViT-B/16 vision features (openai/clip-vit-base-patch16) with a gated adapter and a grounding loss.
- OCR input: OCR text with box, detection, recognition and character features, fed as a block of the encoder input.
- Consformer: self-attention with a constituent prior over OCR words, and a span loss.
- DOVE: training on two views of the OCR block, the original and a "related" copy in which a few OCR words are edited.
- OACT (OCR Adversarial Contrastive Training): a training strategy, not an inference module. Each OCR view gets a noise step
along the sign of its answer-loss gradient (epsilon 0.05 of the token RMS), and a joint supervised-contrastive loss ties the four views
(two clean, two perturbed; separate projection heads for clean and perturbed). It is switched on in this checkpoint
(
dove_adv=ocr_both) and unused at inference.
Results (ViOCRVQA)
Test: 18,601 questions, the full test split (ground truth provided separately; the public test JSON has placeholder answers). Main metric: EM / F1. Scored with training/benchmark_eval.py of the code repository
(the benchmark protocol: EM, F1, BLEU, CIDEr, BERTScore with mBERT). Dev is the validation split at the selected checkpoint.
| Metric | Test | Dev |
|---|---|---|
| Exact Match (EM) | 61.68 | 61.73 |
| F1 (token) | 79.50 | 79.39 |
| F1 (char) | 87.14 | - |
| BLEU-1 | 80.91 | 81.19 |
| BLEU-4 | 73.54 | 73.65 |
| BLEU-avg | 77.16 | - |
| CIDEr | 6.1498 | 6.1934 |
| BERTScore F1 | 93.31 | - |
Single training run (one seed); the checkpoint was selected on the dev set, never on test.
Usage
The architecture is custom (model_type: openvivqa), so it does not load with AutoModel. Use the code at commit
0907f9a, which is the code this checkpoint was trained with, and follow the notebook
notebooks/twc_colab.ipynb (cell 6b, "predict from a Hugging Face repo"). In short:
git clone https://github.com/Kussssssss/ViSceT5 && cd ViSceT5 && git checkout 0907f9a
bash setup.sh
huggingface-cli download Kus669/ViSceT5-ViOCRVQA --local-dir ckpt
MOCK_TEST=false STAGE=predict PREDICT_SPLIT=both PREDICT_DATASET=ViOCRVQA \
PREDICT_CKPT_DIR=ckpt HF_REPO= python3 run_pipeline.py
The script prepares the dataset and its OCR features, generates answers with beam search (4 beams, up to 56 new tokens) and writes a submission CSV.
Limitations
- Trained and evaluated on ViOCRVQA only; no claim is made for other Vietnamese VQA data or other languages.
- The answer depends on the OCR tokens: errors or missing text in the OCR output limit what the model can answer.
- Results are from a single seed.
Citation
A paper describing ViSceT5 is in preparation; the citation will be added here.
- Downloads last month
- 15
Model tree for Kus669/ViSceT5-ViOCRVQA
Base model
VietAI/vit5-base