README.md

---
library_name: transformers
license: apache-2.0
base_model: google/pix2struct-docvqa-base
tags:
- generated_from_trainer
- invoice-processing
- information-extraction
- czech-language
- document-ai
- multimodal-model
- generative-model
- synthetic-data
- hybrid-data
metrics:
- f1
model-index:
- name: Pix2StructCzechInvoice-V2
  results: []
---

# Pix2StructCzechInvoice (V2 – Synthetic + Random Layout + Real Layout Injection)

This model is a fine-tuned version of [google/pix2struct-docvqa-base](https://huggingface.co/google/pix2struct-docvqa-base) for structured information extraction from Czech invoices.

It achieves the following results on the evaluation set:
- Loss: 0.2521  
- F1: 0.7311  

---

## Model description

Pix2StructCzechInvoice (V2) represents an advanced stage of the generative document understanding pipeline.

The model:
- processes full document images  
- generates structured outputs as text sequences  

It is trained to extract key invoice fields:
- supplier  
- customer  
- invoice number  
- bank details  
- totals  
- dates  

This version introduces **real layout injection**, significantly improving visual realism and model generalization.

---

## Training data

The dataset consists of three components:

1. **Synthetic template-based invoices**  
2. **Synthetic invoices with randomized layouts**  
3. **Hybrid invoices with real layouts and synthetic content**  

### Real layout injection

In the hybrid dataset:
- real invoice layouts are used as templates  
- original content is replaced with synthetic data  
- new content is rendered into realistic visual structures  

This preserves:
- real-world layout complexity  
- visual patterns and formatting  
- document structure variability  

while maintaining:
- full control over annotations  
- consistent output format  

---

## Role in the pipeline

This model corresponds to:

**V2 – Synthetic + layout augmentation + real layout injection**

It is used to:
- reduce the domain gap between synthetic and real documents  
- evaluate the effect of realistic layouts on generative models  
- compare with:
  - V0–V1 (synthetic-only training)  
  - V3 (real data fine-tuning)  

---

## Intended uses

- End-to-end invoice extraction from images  
- Document VQA-style tasks  
- Research in generative document understanding  
- Evaluation of hybrid training strategies  

---

## Limitations

- Generated outputs may contain formatting errors  
- Sensitive to decoding strategy and tokenization  
- Still lacks full exposure to real linguistic variability  
- Training remains less stable than classification-based models  

---

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0001
- train_batch_size: 8
- eval_batch_size: 1
- seed: 42
- optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: cosine_with_restarts
- lr_scheduler_warmup_steps: 0.1
- num_epochs: 10
- mixed_precision_training: Native AMP

---

### Training results

| Training Loss | Epoch | Step | Validation Loss | F1     |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 0.3432        | 1.0   | 115  | 0.2771          | 0.6644 |
| 0.1942        | 2.0   | 230  | 0.2611          | 0.6745 |
| 0.1934        | 3.0   | 345  | 0.2521          | 0.7311 |
| 0.1325        | 4.0   | 460  | 0.2665          | 0.7133 |
| 0.1131        | 5.0   | 575  | 0.2686          | 0.6762 |
| 0.1125        | 6.0   | 690  | 0.2601          | 0.7277 |
| 0.1011        | 7.0   | 805  | 0.2962          | 0.7118 |
| 0.1229        | 8.0   | 920  | 0.2893          | 0.7095 |
| 0.0861        | 9.0   | 1035 | 0.3019          | 0.6931 |
| 0.0860        | 10.0  | 1150 | 0.3167          | 0.7186 |

---

## Framework versions

- Transformers 5.0.0  
- PyTorch 2.10.0+cu128  
- Datasets 4.0.0  
- Tokenizers 0.22.2