| <!DOCTYPE html> |
| <html> |
| <head> |
| <meta charset="utf-8"> |
| <meta name="description" content="JQL: Judging Quality across Languages - A pipeline for multilingual data filtering."> |
| <meta name="viewport" content="width=device-width, initial-scale=1"> |
| <title>JQL: Judging Quality across Languages</title> |
| <link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro" rel="stylesheet"> |
| <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/bulma@0.9.4/css/bulma.min.css"> |
| <style> |
| body { font-family: 'Noto Sans', sans-serif; } |
| .hero.is-primary { background-color: #f9d5e5; } |
| .subtitle img { max-width: 100%; height: auto; } |
| .section-title { margin-top: 2em; } |
| </style> |
| </head> |
| <body> |
| <section class="hero is-primary"> |
| <div class="hero-body"> |
| <div class="container has-text-centered"> |
| <h1 class="title is-1">π¦ JQL: Judging Quality across Languages</h1> |
| <p class="subtitle is-5">Scalable and lightweight multilingual data filtering with LLM-based annotators</p> |
| </div> |
| </div> |
| </section> |
|
|
| <section class="section"> |
| <div class="container content"> |
| <p> |
| High-quality multilingual data is crucial for training effective large language models (LLMs). |
| <strong>JQL (Judging Quality across Languages)</strong> is a scalable and lightweight multilingual data filtering approach that distills the judgment capabilities of strong |
| multilingual LLMs into efficient cross-lingual annotators. |
| </p> |
| <p> |
| Overall, JQL improves data quality, retains more tokens, and generalizes to unseen languages. It outperforms heuristic baselines and enables cost-efficient multilingual pretraining data curation at scale. |
| </p> |
| </div> |
| </section> |
| |
| <section class="section"> |
| <div class="container content"> |
| <h2 class="title is-3">𧩠Main Pipeline Steps</h2> |
| <figure> |
| <img src="https://cdn-uploads.huggingface.co/production/uploads/64bfc4d55ce3d382c05c0f9a/1zPQcwqt9Li_gCvd04_2_.png" alt="JQL Pipeline Overview"> |
| <figcaption><em>Figure 1: Overview of the JQL pipeline</em></figcaption> |
| </figure> |
|
|
| <ol> |
| <li><strong>π Ground Truth Creation:</strong> Human annotators label monolingual documents based on a structured instruction prompt. These documents are translated into all target languages to create a multilingual gold-standard dataset. (See Figure 1)</li> |
| <li><strong>π€ LLM-as-a-Judge Selection & Data Annotation:</strong> Strong multilingual LLMs (e.g., Gemma, Mistral, LLaMA) are evaluated against the ground truth, and top-performing models are used to produce synthetic annotations. (See Figure 1)</li> |
| <li><strong>πͺΆ Lightweight Annotator Training:</strong> Train compact regression heads on frozen multilingual embeddings to create efficient, high-throughput annotators. (See Figure 1)</li> |
| <li><strong>π Scalable Data Filtering:</strong> Use trained annotators to filter large-scale pretraining corpora using quantile thresholds. (See Figure 1)</li> |
| </ol> |
| </div> |
| </section> |
|
|
| <section class="section"> |
| <div class="container content"> |
| <h2 class="title is-3">π Results</h2> |
| <ul> |
| <li><strong>βοΈ Accuracy:</strong> Spearmanβs Ο > 0.87 with human ground truth</li> |
| <li><strong>π Downstream LLM Training:</strong> |
| <ul> |
| <li>+7.2% benchmark performance improvement</li> |
| <li>+4.8% token retention vs. FineWeb2 heuristic filter</li> |
| <li>Effective threshold strategies: 0.6 and 0.7 quantile</li> |
| </ul> |
| </li> |
| <li><strong>β‘ Annotation Speed:</strong> ~11,000 docs/min (A100 GPU, avg. 690 tokens)</li> |
| </ul> |
| </div> |
| </section> |
|
|
| <section class="section"> |
| <div class="container content"> |
| <h2 class="title is-3">π Available Artifacts</h2> |
| <ul> |
| <li>π Ground truth annotations in 35 languages</li> |
| <li>π§ Synthetic LLM-annotated dataset (14M+ documents)</li> |
| <li>πͺΆ Lightweight annotation models: |
| <ul> |
| <li>JQL-Gemma</li> |
| <li>JQL-Mistral</li> |
| <li>JQL-Llama</li> |
| </ul> |
| </li> |
| <li>π οΈ Training & inference scripts (coming soon)</li> |
| </ul> |
| </div> |
| </section> |
|
|
| <section class="section"> |
| <div class="container content"> |
| <h2 class="title is-3">π Citation</h2> |
| <p>If you use JQL, the annotations, or the pretrained annotators, please cite the paper:</p> |
| <pre><code>@article{your2024jql, |
| title={JQL: Judging Quality across Languages}, |
| author={Your, Name and Collaborators, Here}, |
| journal={Conference or preprint archive}, |
| year={2024} |
| }</code></pre> |
| </div> |
| </section> |
|
|
| </body> |
| </html> |
