Instructions to use Kus669/ViSceT5-OpenViVQA with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Kus669/ViSceT5-OpenViVQA with Transformers:
# Use a pipeline as a high-level helper # Warning: Pipeline type "visual-question-answering" is no longer supported in transformers v5. # You must load the model directly (see below) or downgrade to v4.x with: # pip install "transformers<5.0.0" from transformers import pipeline pipe = pipeline("visual-question-answering", model="Kus669/ViSceT5-OpenViVQA")# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("Kus669/ViSceT5-OpenViVQA", device_map="auto") - Notebooks
- Google Colab
- Kaggle
ViSceT5-OpenViVQA
ViSceT5 is a Vietnamese scene-text visual question answering model: a ViT5 encoder-decoder that reads the image, the question and the OCR tokens of the image, and generates the answer. This checkpoint is the full model fine-tuned on OpenViVQA.
Model overview
- Backbone: VietAI/vit5-base (encoder-decoder), about 346M parameters in total (fp32, 1.38 GB).
- QA-ViT: question-aware CLIP ViT-B/16 vision features (openai/clip-vit-base-patch16) with a gated adapter and a grounding loss.
- OCR input: OCR text with box, detection, recognition and character features, fed as a block of the encoder input.
- Consformer: self-attention with a constituent prior over OCR words, and a span loss.
- DOVE: training on two views of the OCR block, the original and a "related" copy in which a few OCR words are edited.
- OACT (OCR Adversarial Contrastive Training): a training strategy, not an inference module. Each OCR view gets a noise step
along the sign of its answer-loss gradient (epsilon 0.05 of the token RMS), and a joint supervised-contrastive loss ties the four views
(two clean, two perturbed; separate projection heads for clean and perturbed). It is switched on in this checkpoint
(
dove_adv=ocr_both) and unused at inference.
Results (OpenViVQA)
Test: 3,536 questions, the subset of the test split with public ground truth. Main metric: CIDEr (answers are free-form). Scored with training/benchmark_eval.py of the code repository
(the benchmark protocol: EM, F1, BLEU, CIDEr, BERTScore with mBERT). Dev is the validation split at the selected checkpoint.
| Metric | Test | Dev |
|---|---|---|
| Exact Match (EM) | 13.21 | 13.71 |
| F1 (token) | 58.76 | 58.59 |
| F1 (char) | 69.36 | - |
| BLEU-1 | 55.52 | 55.23 |
| BLEU-4 | 41.23 | 41.09 |
| BLEU-avg | 47.85 | - |
| CIDEr | 3.7202 | 3.7568 |
| BERTScore F1 | 85.85 | - |
Single training run (one seed); the checkpoint was selected on the dev set, never on test.
Usage
The architecture is custom (model_type: openvivqa), so it does not load with AutoModel. Use the code at commit
0907f9a, which is the code this checkpoint was trained with, and follow the notebook
notebooks/twc_colab.ipynb (cell 6b, "predict from a Hugging Face repo"). In short:
git clone https://github.com/Kussssssss/ViSceT5 && cd ViSceT5 && git checkout 0907f9a
bash setup.sh
huggingface-cli download Kus669/ViSceT5-OpenViVQA --local-dir ckpt
MOCK_TEST=false STAGE=predict PREDICT_SPLIT=both PREDICT_DATASET=OpenViVQA \
PREDICT_CKPT_DIR=ckpt HF_REPO= python3 run_pipeline.py
The script prepares the dataset and its OCR features, generates answers with beam search (4 beams, up to 56 new tokens) and writes a submission CSV.
Limitations
- Trained and evaluated on OpenViVQA only; no claim is made for other Vietnamese VQA data or other languages.
- The answer depends on the OCR tokens: errors or missing text in the OCR output limit what the model can answer.
- Results are from a single seed.
Citation
A paper describing ViSceT5 is in preparation; the citation will be added here.
- Downloads last month
- 53
Model tree for Kus669/ViSceT5-OpenViVQA
Base model
VietAI/vit5-base