whybe-choi/kovre-stage1

KoVRE: Korean Visual Document Retrieval Embedding

KoVRE-Stage1 is a 2B-parameter single-vector embedding model for Korean visual document retrieval. It directly matches text queries against rendered document-page images and serves as the contrastive-learning checkpoint used before the knowledge-distillation stage of KoVRE.

The model is initialized from Qwen/Qwen3-VL-Embedding-2B and trained on 708,729 Korean and English query-page pairs using positive-aware hard-negative mining, self-guide filtering, hardness weighting, in-batch negatives, and Matryoshka Representation Learning.

For the strongest Korean VDR performance, use the final whybe-choi/kovre checkpoint, which further applies reranker-based knowledge distillation.

• Code: https://github.com/whybe-choi/kovre
• Paper: WIP

Highlights

• Bilingual visual document retrieval: Trained on 406,945 Korean and 301,784 English query-page pairs.
• Efficient single-vector representation: Stores one dense vector per document page.
• Positive-aware hard negatives: Uses seven hard negatives per query mined with Qwen3-VL-Embedding-8B while filtering likely false negatives.
• Robust contrastive training: Combines self-guide filtering, hardness weighting, explicit hard negatives, and in-batch negatives.
• Flexible embedding dimensions: Matryoshka-trained at 128, 256, 512, 768, 1,024, and 2,048 dimensions.
• Strong Korean retrieval performance: Reaches 0.5792 OVR nDCG@10 across KoViDoRe and SDS KoPub VDR before knowledge distillation.

Model Overview

Property	Value
Base model	Qwen/Qwen3-VL-Embedding-2B
Parameters	2B
Training stage	Stage 1: Contrastive learning
Representation	Single vector
Full embedding dimension	2,048
Matryoshka training dimensions	128, 256, 512, 768, 1,024, 2,048
Input modalities	Text query and document-page image
Similarity function	Cosine similarity
Pooling	Last-token pooling
Maximum image resolution used in training	1,280 visual tokens (approximately 1.3M pixels)
Training languages	Korean and English

Intended Use

For production use or the best reported Korean benchmark performance, we recommend whybe-choi/kovre.

KoVRE-Stage1 is intended for:

• Korean visual document retrieval
• OCR-free retrieval over scanned or digitally rendered document pages
• English and Korean text-to-document-image retrieval
• First-stage retrieval for multimodal retrieval-augmented generation
• Research on contrastive learning, hard-negative treatment, and Matryoshka embeddings
• Continued training or domain adaptation before task-specific distillation

Usage

Sentence Transformers

# pip install -U sentence-transformers

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("whybe-choi/kovre-stage1")

queries = [
    "2024년 정보보호 예산은 얼마인가요?",
    "재생에너지 발전량 추이를 보여주는 표",
]
document_pages = [
    "pages/page_1.png",
    "pages/page_2.png",
    "pages/page_3.png",
]

query_embeddings = model.encode(
    queries,
    prompt="Find a document image that matches the given query.",
    normalize_embeddings=True,
)
page_embeddings = model.encode(
    document_pages,
    normalize_embeddings=True,
)

scores = model.similarity(query_embeddings, page_embeddings)
print(scores)

For a smaller index, initialize the model with a Matryoshka dimension:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer(
    "whybe-choi/kovre-stage1",
    truncate_dim=256,
)

The model was trained with the following instructions:

• Query: Find a document image that matches the given query.
• Document page: Represent the user's input. (the default instruction)

Transformers

KoVRE preserves the architecture of Qwen3-VL-Embedding. For lower-level inference with Transformers, use the Qwen3-VL-Embedding inference code and replace the model path with whybe-choi/kovre.

Recommended dependencies:

transformers>=4.57.0
qwen-vl-utils>=0.0.14
torch>=2.8.0

Input and Output

Input

• Query: Korean or English text describing the information to retrieve
• Document: A rendered document-page image
• Recommended query instruction: Find a document image that matches the given query.

Output

• Type: L2-normalized dense vector
• Dimensions: 128 to 2,048; 2,048 by default
• Scoring: Cosine similarity; higher values indicate greater relevance

For Matryoshka inference, truncate the full embedding to the desired prefix dimension and L2-normalize it before computing similarity. The dimensions explicitly optimized during training were 128, 256, 512, 768, 1,024, and 2,048.

Training

Training Data

Language	Query-page pairs
Korean	406,945
English	301,784
Total	708,729

The Korean data combines the public training resource released with KoViDoRe and an additional private Korean collection. The English data combines five visual document retrieval resources covering reports, slides, tables, and other visually structured pages. English examples are included to help preserve the backbone's existing retrieval ability during Korean adaptation.

Hard-Negative Construction

Seven hard negatives are mined per query using Qwen3-VL-Embedding-8B. Within each source dataset, document pages are ranked by cosine similarity after excluding known positives. Candidates scoring above 95% of the annotated positive score are removed to reduce false negatives, and query-positive pairs whose positive score does not exceed 0.3 are filtered out.

Contrastive Objective

Training uses an InfoNCE objective over the paired positive, seven mined hard negatives, and in-batch negatives. The objective includes:

• Self-guide filtering with a threshold of -0.1
• Hardness weighting with alpha = 2
• Matryoshka losses at 128, 256, 512, 768, 1,024, and 2,048 dimensions
• Cosine similarity with normalized embeddings

The full 2B model is trained for one epoch with bfloat16 mixed precision. GradCache is used to support a larger effective contrastive batch under the available GPU memory.

Evaluation

We evaluate KoVRE with nDCG@10 on two Korean visual document retrieval benchmarks:

• KoViDoRe: Four document domains—cybersecurity, economics, energy, and human resources. A query may have multiple relevant pages.
• SDS KoPub VDR: Korean public documents with textual, visual, and cross-modal queries; used as an out-of-distribution evaluation set.

AVG is the mean across the four KoViDoRe domains. OVR is the macro-average across those four domains and SDS KoPub VDR.

Model	Params	KoViDoRe					SDSKoPub	OVR
		Cybersecurity	Economic	Energy	HR	AVG
jina-clip-v2	0.9B	0.1993	0.0011	0.1096	0.0294	0.0849	0.0732	0.0825
jina-v5-omni-nano	1B	0.4404	0.0640	0.2017	0.0695	0.1939	0.0961	0.1743
jina-v5-omni-small	2B	0.4357	0.0744	0.2380	0.1029	0.2128	0.1902	0.2082
Qwen3-VL-Embedding-2B	2B	0.6111	0.1592	0.4123	0.1842	0.3417	0.4285	0.3591
Qwen3-VL-Embedding-8B	8B	0.7809	0.2373	0.6360	0.3613	0.5039	0.7293	0.5489
jina-embeddings-v4 (single-vector)	4B	0.7280	0.2058	0.6273	0.4106	0.4929	0.7222	0.5388
jina-embeddings-v4 (multi-vector)	4B	0.7714	0.2359	0.6752	0.4799	0.5406	0.7802	0.5885
KoVRE (Stage 1)	2B	0.7444	0.2797	0.6506	0.5002	0.5437	0.7214	0.5792
KoVRE (Stage 1 + Stage 2)	2B	0.7627	0.2987	0.6576	0.5082	0.5568	0.7324	0.5919

License

KoVRE is released under the Apache 2.0 License.

Citation

If you find KoVRE useful, please cite:

@misc{choi2026kovre,
  title        = {KoVRE: Training an Efficient Embedding Model for Korean Visual Document Retrieval},
  author       = {Yongbin Choi and Gyuho Shim and Youngjoon Jang},
  year         = {2026}
}