🌸 Stable Benchmark jp-JP — Beta Version

We are excited to introduce the Beta Version of our jp-JP Stable Benchmark, an expansion of our G11n GenAI Assessment Model into the Japanese locale.

This benchmark adapts our existing Stable Benchmark methodology to Japanese language and culture, while preserving the same core quality assessment approach used for our previous locale evaluations.

---

🎯 What is the jp-JP Stable Benchmark?

The jp-JP Stable Benchmark is a curated set of 130 prompts designed to evaluate how AI models handle Japanese language, cultural context, instructions, and multimodal scenarios.

The benchmark follows the same four quality dimensions used in our G11n assessment framework:

• 💬 Language & Grammar
• 📄 Instruction & Response Coherence
• 🌏 Cultural Adaptation
• 🖼️ Multimodal Consistency

However, the benchmark is not simply a translation of the existing prompts. The content has been adapted to account for characteristics that are specific to Japanese, including linguistic structures, expressions, cultural references, and locale-specific scenarios.

---

📊 Benchmark Structure

The 130 prompts are distributed across the four evaluation dimensions:

Evaluation Dimension	Prompts
Language & Grammar	30
Cultural Adaptation	20
Instruction & Response Coherence	30
Multimodal Consistency	50
Total	130

The benchmark also includes different prompt types such as open-ended and closed-ended questions, translations, comparisons, calculations, judgements, image generation, audio generation, file generation, and code generation.

---

🈶 Japanese-Specific Language & Grammar

The Language & Grammar section was adapted to evaluate aspects that are particularly relevant to Japanese rather than treating Japanese as a direct translation target.

Examples include:

• Differences between Hyōjungo (standard Japanese) and regional varieties such as Kansai-ben.
• Japanese homophones and words with different meanings depending on their kanji and context.
• Correct use and ordering of Japanese grammatical structures.
• Preservation of meaning when translating idiomatic expressions.
• Spelling, punctuation, syntax, and semantic accuracy in Japanese responses.

For example, the benchmark includes prompts that test whether a model can distinguish between Japanese words with the same pronunciation but different meanings, such as 「神」「紙」「髪」, and whether it can correctly interpret the meaning from context.

It also includes comparisons between standard Japanese and regional expressions, helping us evaluate whether a model recognizes differences in usage, meaning, and nuance rather than treating all Japanese as linguistically uniform.

---

🧪 Why a Beta Version?

This release is intentionally being introduced as a Beta Version. The goal is to validate the benchmark before establishing the jp-JP dataset as a stable reference for future regression evaluations.

During this phase, we will review the benchmark from both a linguistic and QA perspective, looking for issues such as:

• Prompt ambiguity or unintended interpretations.
• Incorrect or unnatural Japanese phrasing.
• Expected results that require refinement.
• Cultural references that may need additional context.
• Success Criteria that may not adequately capture Japanese-specific behavior.
• Multimodal scenarios that require adjustment for the Japanese locale.

Feedback gathered during the Beta phase will help us refine the dataset before the jp-JP Stable Benchmark Release.

---

🔄 From Beta to Stable

The intended workflow is:

Beta Benchmark → Linguistic & QA Review → Refinement → Stable Benchmark → Regression Evaluation

Once the benchmark has completed its review and refinement stage, the final jp-JP Stable Benchmark can be used as a consistent reference for evaluating future model versions and identifying changes in Japanese globalization quality over time.

---

🌏 Expanding Globalization Coverage

The jp-JP benchmark represents the next step in expanding our G11n evaluation coverage beyond the original locale.

Our goal is not simply to translate an existing benchmark into additional languages, but to build evaluation datasets that reflect the linguistic, cultural, and contextual characteristics of each locale.

This approach allows us to evaluate whether AI models can provide experiences that feel appropriate for users in different regions—not only whether they can produce grammatically correct translations.

---

📚 Explore the Benchmark

The jp-JP Stable Benchmark Beta is part of the DilatoMX G11n GenAI Assessment Model .

We welcome feedback from linguists, localization professionals, QA engineers, AI evaluators, and Japanese-language experts as we continue refining the benchmark.

🇯🇵 New language. Broader perspectives. Same commitment to quality.

---

Dilato MX — Globalization & Quality Engineering