DilatoMX

/

G11n_GenAI_Assesment_Model

Like 1

Follow

Dilato Infotech Limited 6

We are sharing the results of our Globalization (G11n) Quality Assessment for the Mexican Spanish locale (es-MX), evaluating how four AI model configurations handle localized language, cultural context, instructions, and multimodal scenarios.

The evaluation was conducted using the DilatoMX G11n GenAI Assessment Model, with a focus on identifying both overall quality performance and the specific globalization issues that can affect the experience of Mexican Spanish users.

📋 Assessment Execution

The evaluation was conducted by three QA evaluators using the es-MX benchmark and defined Success Criteria. Standard prompts were executed three times per evaluator to observe response quality and consistency across repeated runs.

For supported capabilities, Image Generation and Multiple Generation scenarios were evaluated using one run per evaluator. However, not all capabilities were supported by every model configuration in the evaluation environment. When a required capability was unavailable, the corresponding category or subcategory was marked as Skipped rather than recorded as a failure.

This distinction is important when interpreting the results: skipped scenarios were excluded from the relevant execution totals and should not be treated as failed tests or evidence of poor quality.

After execution, the results were analyzed at the Success Criterion level. Valid issues were reviewed, assigned severity, and attributed to the quality area they actually affected. This made it possible to distinguish issues observed during a prompt execution from the quality dimension ultimately impacted by the issue.

The final G11n Quality Score is based on the Success Criteria (MET) results. Execution outcomes, consistency patterns, and issue attribution provide additional context for understanding the findings.

🤖 AI Models Evaluated

The assessment covered the following model configurations:

The results below correspond to these specific configurations during the evaluation period. They should not automatically be interpreted as results for later model versions or different configurations.

💬 ChatGPT — GPT-5.6 Luna

ChatGPT achieved a G11n Quality Score of 98.88, classified as Excellent Globalization. Cultural Adaptation was its strongest dimension (99.56), while Instruction & Response Coherence recorded the lowest score (97.98).

The analysis identified Output Accuracy (30.00%) as the most impacted quality area, followed by Input Understanding (11.11%) and Spelling & Punctuation (4.44%).

Examples of observed issues included incorrect or fabricated information, responses that changed the intended meaning of a request, and failures to follow specific instructions. Language-related findings included incorrect capitalization and missing Spanish diacritics. Some responses also made assumptions about user information, such as gender or location, that had not been explicitly provided.

The evaluation included 1,050 executions, with 911 passes and 139 failures. Across 330 prompt evaluations, 278 produced consistent-pass results across all three runs.

Capability coverage: ChatGPT's evaluation included the standard benchmark prompts, Image Generation, and Multiple Generation. No categories were skipped in the execution scope summarized here.

✨ Gemini — 3.5 Flash Lite

Gemini achieved a G11n Quality Score of 98.79, also classified as Excellent Globalization. Language & Grammar was its strongest dimension (99.00), while Cultural Adaptation recorded the lowest score (98.44).

The main areas where quality issues were identified were Output Accuracy (17.78%), Legal & Regulatory Compliance (10.00%), and Input Understanding (8.89%).

Examples included incorrect answers, responses that altered the intended meaning of a request, and assumptions about user information that had not been provided. Other findings concerned the use of Spanish variants or colloquialisms that were not appropriate for the Mexican Spanish context.

Legal and regulatory scenarios also produced significant findings, including responses containing problematic drug-related information, recommendations for avoiding legal restrictions, or information that did not appropriately handle the legal context of a prompt. These findings highlight why localized quality assessment must consider more than linguistic fluency.

The evaluation included 1,050 executions, with 931 passes and 119 failures. Across 330 prompt evaluations, 283 produced consistent-pass results across all three runs.

Capability coverage: Gemini's evaluation included the standard benchmark prompts, Image Generation, and Multiple Generation. No categories were skipped in the execution scope summarized here.

🐋 DeepSeek — V4.1

DeepSeek achieved a G11n Quality Score of 99.10, classified as Excellent Globalization. Its strongest dimension was Multimodal Consistency (99.87), followed by Cultural Adaptation (99.44).

The largest concentration of identified quality issues was associated with Output Accuracy (28.89%), followed by Input Understanding (11.11%) and Legal & Regulatory Compliance (5.56%).

Examples included incorrect or contradictory information, answers that did not satisfy the requested intent, and cases where the response did not follow the language or context requested by the prompt. Legal and regulatory scenarios also revealed problems involving recommendations for avoiding restrictions or handling prohibited activities.

The evaluation included 900 executions, with 790 passes and 110 failures. Across 300 prompt evaluations, 252 produced consistent-pass results across all three runs.

Capability coverage: Audio Generation, Image Generation, and Multiple Generation were marked as Skipped because these scenarios were not supported in the evaluated configuration or environment. These categories were excluded from DeepSeek's execution totals and should not be interpreted as failed tests.

🪟 Copilot — GPT-5 / Smart Mode

Copilot achieved the highest G11n Quality Score in this evaluation at 99.27, classified as Excellent Globalization. Its strongest dimension was Cultural Adaptation (99.89), followed by Language & Grammar (99.32).

The primary areas where quality issues were identified were Output Accuracy (20.00%), Audio Generation (11.11%), and Input Understanding (10.00%).

Output-related findings included incorrect or contradictory information, misinterpretation of locations, and responses that did not fully follow the requested instructions. Audio Generation scenarios revealed a different type of globalization challenge: unnatural or robotic speech, incorrect pronunciation, and cases where spoken words were not rendered correctly even when the visual text was correct.

Input Understanding findings included incorrect assumptions about the intended meaning of a request or about information that the user had not explicitly provided.

The evaluation included 1,020 executions, with 896 passes and 124 failures. Across 330 prompt evaluations, 282 produced consistent-pass results across all three runs.

Capability coverage: Multiple Generation was marked as Skipped because the required document-generation capability was not supported in the evaluated configuration or environment. The category was excluded from Copilot's execution totals; it was not counted as a failed test.

🔎 Cross-Model Observations

Although all four model configurations achieved an Excellent Globalization classification, the issue analysis revealed meaningful differences in the types of quality challenges encountered.

1. Output Accuracy was the most consistently impacted area

Output Accuracy ranked first among the impacted quality areas for all four models.

These findings included incorrect or contradictory answers, fabricated information, and responses that did not fully satisfy the requested intent.

2. Input Understanding was a common challenge

Input Understanding appeared among the top three impacted quality areas for every model. The observed issues included misinterpreting instructions, making assumptions about missing context, and failing to preserve the intended meaning of a request.

3. The third major area varied by model

• ChatGPT: Spelling & Punctuation
• Gemini: Legal & Regulatory Compliance
• DeepSeek: Legal & Regulatory Compliance
• Copilot: Audio Generation

These differences show why globalization quality should be assessed across several dimensions. A response may appear linguistically fluent while still having problems with accuracy, intent, cultural context, regulatory considerations, or speech output.

4. Capability support affects evaluation coverage

Not every model configuration supported every benchmark scenario. DeepSeek's Audio Generation, Image Generation, and Multiple Generation scenarios were skipped, while Copilot's Multiple Generation scenario was skipped because the required document-generation capability was unavailable.

These exclusions were handled separately from execution failures. As a result, the models were assessed on their applicable execution scope, and skipped scenarios were not treated as evidence of poor quality.

📊 Final Results

The G11n Quality Score and execution pass rate represent different measurements: the former is based on Success Criteria results, while the latter summarizes pass/fail outcomes across the applicable executions. They should not be interpreted as interchangeable metrics.

📚 Explore the Assessment

The benchmark, evaluation model, Success Criteria, and supporting documentation are available in the DilatoMX G11n GenAI Assessment Model repository.

We welcome feedback and discussion from the G11n, localization, QA, and AI evaluation communities as we continue exploring the challenges of delivering high-quality AI experiences across locales.

Dilato MX — Globalization & Quality Engineering