{"title": "PUMA: Polish Unified Multimodal Assessment", "description": "We present PUMA (Polish Unified Multimodal Assessment), a benchmark designed to evaluate the linguistic and cultural competencies of large models in the context of Poland and the Polish language. PUMA consists of 900 hand-crafted questions designed to assess a wide spectrum of multimodal competencies. It is divided into three modalities: images, audio, and documents. Each modality contains three categories targeting distinct competencies or knowledge domains. Therefore, there are nine categories in total, with 100 questions per category. Six of these categories focus on question answering (QA), evaluating the models' knowledge and reasoning abilities in multimodal contexts. We also included three specialized categories testing the models' practical competencies in typical vision and audio tasks: automatic speech recognition (ASR), optical character recognition (OCR), and document structure extraction. The benchmark uses deterministic, rule-based verification process, without external judge.
We evaluated models that support all required modalities, as well as VLMs that support only image and document categories. Results for multimodal models and pairs of models from the same family (e.g. GPT-5.5 + GPT-Audio) are available in the Full benchmark. Results for vision models are available in the Vision only benchmark.", "categories": "\n\n\n \n \n \n \n\n\n \n \n \n \n\n\n \n \n \n \n\n\n

Images

History and culture - This category tests the models' knowledge regarding the history, tradition, cultural heritage, and customs of Poland. It also includes art, primarily painting, sculpture, and architecture. Some of the questions concern people or objects of significant historical importance.Contemporary life - It covers contemporary life and pop culture. Some of the questions involve modern media such as film, television, and the internet. We have also included questions related to sports, contemporary politics, and show business. Cultural and social issues are also addressed, provided they relate to contemporary topics.Geography and environment - This category verifies the models' knowledge of Poland's geography, including its fauna, flora, and well-known natural landmarks. Questions about man-made structures may also be found in this category. These are primarily related to infrastructure, cities, as well as administrative and socioeconomic issues.

Audio

Automatic Speech Recognition (ASR) - This is a specific category designed to test models' ability to transcribe speech, a task of significant practical importance. The recordings used in this category range from about a minute to several minutes in length. For the most part, these are challenging samples containing interference, background noise, complex language, or dialects.Speech QA - It evaluates advanced speech understanding and knowledge of the Polish cultural context based on audio recordings. This category includes questions about short audio samples lasting from about a minute to several minutes. The scope of tested competencies covers speech comprehension, information extraction, recognition of the speakers' intentions and emotions, as well as the ability to link the content of the recording with broader knowledge about Poland.Sound and music QA - This category includes questions about audio recordings where the main content is not speech, but other sounds such as ambient noises, musical instruments, animals, jingles, and the sounds of tools and machinery. All recordings are related to Poland, and the models' task is to recognize the sounds and use broader knowledge to interpret their content.

Documents

Optical character recognition (OCR) - This category evaluates the models' ability to convert documents from image formats into plain text. The tasks mostly involve challenging cases, such as handwritten text or low-quality scans. Additionally, some examples contain tables. The metric we use verifies the correct extraction of tables while preserving their structure.Document QA - This category tests the models' ability to understand and interpret documents in Polish by answering questions about their content. Most of the examples in this category consist of visually rich documents that require the analysis of text, tables, charts, or infographics, among others.Structured extraction - This category evaluates the models' ability to extract relevant information from documents and save it in a structured JSON format. Each example in this category includes a document in the form of an image and defines a JSON schema that specifies the expected format of the model's response.
", "grading": "Questions are evaluated using verification rules. For each question, one or multiple such rules can be defined. A single rule verifies the alignment of the model's response with user-defined correctness criteria. To standardize the evaluation framework across tasks and rule types, we adopted a binary primary metric, meaning the model receives either zero or one point for its response. We refer to this as the strict score. Furthermore, for each rule, we defined a soft score, which yields fractional values if the model's response at least partially satisfies the predefined criteria. Since soft score distributions vary across different rule types, these values should be interpreted only as an auxiliary metric. If a question is associated with multiple verification rules, the response is deemed correct only if all rules are satisfied. The final score on the benchmark is calculated as the mean of the strict scores, representing the percentage of correct responses.\n
\nOur evaluation framework supports seven types of verification rules. Four of them are applicable to categories related to reasoning and question answering. Additionally, we introduced three specialized metrics to evaluate categories corresponding to other task types.\n
\nKnowledge-based categories - Questions in the six knowledge and reasoning categories are graded using deterministic, rule-based verification. Annotators specify which words or phrases must be included, which must be excluded, and, where necessary, their required order or a regular-expression pattern. Responses are normalized by removing punctuation, converting text to lowercase, and lemmatizing words. Each question receives a binary strict score: if multiple rules are assigned, all of them must be satisfied to obtain one point. Partial compliance is reflected in an auxiliary soft score, usually through the proportion of correctly matched phrases.\n
\nSpecial categories (ASR, OCR, and Structured Extraction) - These categories use task-specific metrics. ASR is evaluated with word accuracy, defined as WAcc=max(0,1-WER), after case-insensitive normalization that ignores punctuation and other non-alphanumeric characters. Its soft score equals WAcc, while a strict point is awarded when WAcc reaches at least 0.9. OCR combines WAcc for ordinary text with TEDS, which assesses both the content and structure of extracted HTML tables. These components are weighted according to the amount of textual and tabular content, and the resulting value is used as the soft score; a score of at least 0.9 is required for strict success. Structured extraction requires valid JSON conforming to the supplied schema and to the provieded reference answer. Its soft score is the proportion of fields whose values match the reference object.", "paper": "https://arxiv.org/abs/2608.21853", "citation": "@misc{dadas2026puma,\n title={PUMA: A Polish Benchmark for Culturally Grounded Multimodal Understanding}, \n author={S{\\l}awomir Dadas and Micha{\\l} Pere{\\l}kiewicz and Rafa{\\l} Po{\\'s}wiata and Ma{\\l}gorzata Gr\\k{e}bowiec and Bart{\\l}omiej Jaworski and Izabela Wo{\\'z}niakowska},\n year={2026},\n eprint={2608.21853},\n archivePrefix={arXiv},\n primaryClass={cs.CL},\n url={https://arxiv.org/abs/2608.21853}, \n}", "options": {"closeable": false, "expandable": true, "showHelp": false, "showFooter": false, "showChangelog": true, "average": true, "tasksWon": false}, "columns": [{"name": "Provider", "format": "nominal"}], "metrics": [{"id": "Strict", "name": "Strict", "greaterIsBetter": true}, {"id": "Soft", "name": "Soft", "greaterIsBetter": true}], "taskGroups": [{"id": "Images", "name": "Images", "expanded": true}, {"id": "Audio", "name": "Audio", "expanded": true}, {"id": "Documents", "name": "Documents", "expanded": true}], "tasks": [{"id": "History and culture", "name": "History &\nculture", "groupId": "Images", "precision": 2}, {"id": "Contemporary life", "name": "Contemporary\nlife", "groupId": "Images", "precision": 2}, {"id": "Geography and environment", "name": "Geography &\nenvironment", "groupId": "Images", "precision": 2}, {"id": "ASR", "name": "ASR", "groupId": "Audio", "precision": 2}, {"id": "Speech QA", "name": "Speech QA", "groupId": "Audio", "precision": 2}, {"id": "Sound and music QA", "name": "Sound &\nmusic QA", "groupId": "Audio", "precision": 2}, {"id": "OCR", "name": "OCR", "groupId": "Documents", "precision": 2}, {"id": "Document QA", "name": "Document QA", "groupId": "Documents", "precision": 2}, {"id": "Structured extraction", "name": "Structured\nextraction", "groupId": "Documents", "precision": 2}], "models": [{"id": "bielik-0.1", "name": "Bielik-0.1", "url": "https://huggingface.co/speakleash/Bielik-7B-Instruct-v0.1"}, {"id": "bielik-2.1", "name": "Bielik-2.1", "url": "https://huggingface.co/speakleash/Bielik-11B-v2.1-Instruct"}, {"id": "bielik-2.2", "name": "Bielik-2.2", "url": "https://huggingface.co/speakleash/Bielik-11B-v2.2-Instruct"}, {"id": "bielik-2.3", "name": "Bielik-2.3", "url": "https://huggingface.co/speakleash/Bielik-11B-v2.3-Instruct"}, {"id": "bielik-2.5", "name": "Bielik-2.5", "url": "https://huggingface.co/speakleash/Bielik-11B-v2.6-Instruct"}, {"id": "bielik-2.6", "name": "Bielik-2.6", "url": "https://huggingface.co/speakleash/Bielik-11B-v2.6-Instruct"}, {"id": "bielik-3-4.5b", "name": "Bielik-4.5B-v3.0-Instruct", "url": "https://huggingface.co/speakleash/Bielik-4.5B-v3.0-Instruct"}, {"id": "bielik-3-1.5b", "name": "Bielik-1.5B-v3.0-Instruct", "url": "https://huggingface.co/speakleash/Bielik-1.5B-v3.0-Instruct"}, {"id": "Bielik-11B-v3.0-Instruct", "name": "Bielik-11B-v3.0-Instruct", "url": "https://huggingface.co/speakleash/Bielik-11B-v3.0-Instruct"}, {"id": "Bielik-Minitron-7B-v3.0-Instruct", "name": "Bielik-Minitron-7B-v3.0-Instruct", "url": "https://huggingface.co/speakleash/Bielik-Minitron-7B-v3.0-Instruct"}, {"id": "llama-3.1-8b", "name": "Llama-3.1-8B", "url": "https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct"}, {"id": "llama-3.1-70b", "name": "Llama-3.1-70B", "url": "https://huggingface.co/meta-llama/Llama-3.1-70B-Instruct"}, {"id": "llama-3.0-70b", "name": "Llama-3.0-70B", "url": "https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct"}, {"id": "llama-3.3-70b", "name": "Llama-3.3-70B", "url": "https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct"}, {"id": "llama-3.1-405b", "name": "Llama-3.1-405b", "url": "https://huggingface.co/meta-llama/Llama-3.1-405B-Instruct"}, {"id": "claude-3-5-haiku-20241022", "name": "Claude-3.5-Haiku-20241022"}, {"id": "claude-3-opus", "name": "Claude-3-Opus"}, {"id": "claude-3.5-sonnet", "name": "Claude-3.5-Sonnet-20241022"}, {"id": "claude-3.5-sonnet-20240620", "name": "Claude-3.5-Sonnet-20240620"}, {"id": "claude-3.7-sonnet", "name": "Claude-3.7-Sonnet"}, {"id": "claude-3.7-sonnet-thinking", "name": "Claude-3.7-Sonnet-Thinking"}, {"id": "claude-3-sonnet", "name": "Claude-3.0-Sonnet"}, {"id": "claude-4-sonnet", "name": "Claude-Sonnet-4"}, {"id": "claude-4-opus", "name": "Claude-Opus-4"}, {"id": "claude-opus-4.1", "name": "Claude-Opus-4.1"}, {"id": "claude-opus-4.5", "name": "Claude-Opus-4.5"}, {"id": "claude-opus-4.6", "name": "Claude-Opus-4.6"}, {"id": "claude-opus-4.7", "name": "Claude-Opus-4.7"}, {"id": "claude-sonnet-4.5", "name": "Claude-Sonnet-4.5"}, {"id": "claude-sonnet-4.6", "name": "Claude-Sonnet-4.6"}, {"id": "claude-haiku-4.5", "name": "Claude-Haiku-4.5"}, {"id": "ministral-3b-2512", "name": "Ministral-3b-2512", "url": "https://huggingface.co/mistralai/Ministral-3-3B-Instruct-2512"}, {"id": "ministral-8b-2512", "name": "Ministral-8b-2512", "url": "https://huggingface.co/mistralai/Ministral-3-8B-Instruct-2512"}, {"id": "ministral-14b-2512", "name": "Ministral-14b-2512", "url": "https://huggingface.co/mistralai/Ministral-3-14B-Instruct-2512"}, {"id": "ministral-8b", "name": "Ministral-8b", "url": "https://huggingface.co/mistralai/Ministral-8B-Instruct-2410"}, {"id": "mistral-7b-v0.3", "name": "Mistral-7b-v0.3", "url": "https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3"}, {"id": "mistral-large-2407", "name": "Mistral-Large-2407", "url": "https://huggingface.co/mistralai/Mistral-Large-Instruct-2407"}, {"id": "mistral-large-2411", "name": "Mistral-Large-2411", "url": "https://huggingface.co/mistralai/Mistral-Large-Instruct-2411"}, {"id": "Mistral-Large-3-2512", "name": "Mistral-Large-3-2512", "url": "https://huggingface.co/mistralai/Mistral-Large-3-675B-Instruct-2512"}, {"id": "mistral-nemo", "name": "Mistral-Nemo", "url": "https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407"}, {"id": "mixtral-8x22b-instruct", "name": "Mixtral-8x22b", "url": "https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1"}, {"id": "mixtral-8x7b-instruct", "name": "Mixtral-8x7b", "url": "https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1"}, {"id": "Mistral-Small-24B-Instruct-2501", "name": "Mistral-Small-24B-2501", "url": "https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501"}, {"id": "mistral-small-3.1-24b-2503", "name": "Mistral-Small-3.1-24B-2503", "url": "https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503"}, {"id": "mistral-small-3.2-24b-2506", "name": "Mistral-Small-3.2-24B-2506", "url": "https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506"}, {"id": "mistral-medium-3", "name": "Mistral-Medium-3"}, {"id": "Mistral-Medium-3.1", "name": "Mistral-Medium-3.1"}, {"id": "Mistral-3.2-24B", "name": "Mistral-3.2-24B", "url": "https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506"}, {"id": "Mistral-Medium-3.5", "name": "Mistral-Medium-3.5", "url": "https://huggingface.co/mistralai/Mistral-Medium-3.5-128B"}, {"id": "Mistral-Small-4", "name": "Mistral-Small-4", "url": "https://huggingface.co/mistralai/Mistral-Small-4-119B-2603"}, {"id": "magistral-small-2506", "name": "Magistral-Small-2506", "url": "https://huggingface.co/mistralai/Magistral-Small-2506"}, {"id": "wizardlm-2-8x22b", "name": "WizardLM-2-8x22b", "url": "https://huggingface.co/alpindale/WizardLM-2-8x22B"}, {"id": "gpt-3.5-turbo", "name": "GPT-3.5-turbo"}, {"id": "gpt-4-turbo", "name": "GPT-4-turbo"}, {"id": "gpt-4o-2024-05-13", "name": "GPT-4o-2024-05-13"}, {"id": "gpt-4o-2024-08-06", "name": "GPT-4o-2024-08-06"}, {"id": "gpt-4o-2024-11-20", "name": "GPT-4o-2024-11-20"}, {"id": "gpt-4o-mini-2024-07-18", "name": "GPT-4o-mini-2024-07-18"}, {"id": "gpt-4", "name": "GPT-4"}, {"id": "gpt-4.5-preview-2025-02-27", "name": "GPT-4.5-preview-2025-02-27"}, {"id": "gpt-4.1-2025-04-14", "name": "GPT-4.1-2025-04-14"}, {"id": "gpt-4.1-mini-2025-04-14", "name": "GPT-4.1-mini-2025-04-14"}, {"id": "gpt-4.1-nano-2025-04-14", "name": "GPT-4.1-nano-2025-04-14"}, {"id": "gpt-5.5-no", "name": "GPT-5.5 (no reasoning)"}, {"id": "gpt-5.5-low", "name": "GPT-5.5 (low reasoning)"}, {"id": "gpt-5.5-high", "name": "GPT-5.5 (high reasoning)"}, {"id": "gpt-5-2025-08-07", "name": "GPT-5-2025-08-07"}, {"id": "gpt-5-mini-2025-08-07", "name": "GPT-5-mini-2025-08-07"}, {"id": "gpt-5-nano-2025-08-07", "name": "GPT-5-nano-2025-08-07"}, {"id": "gpt-5-pro-2025-10-06", "name": "GPT-5-Pro-2025-10-06 (high reasoning)"}, {"id": "gpt-5.1-2025-11-13", "name": "GPT-5.1-2025-11-13 (default reasoning)"}, {"id": "gpt-5.1-2025-11-13-high", "name": "GPT-5.1-2025-11-13 (high reasoning)"}, {"id": "gpt-5.2-2025-12-11", "name": "GPT-5.2-2025-12-11 (medium reasoning)"}, {"id": "gpt-5.2-2025-12-11-high", "name": "GPT-5.2-2025-12-11 (high reasoning)"}, {"id": "gpt-5.2-2025-12-11-xhigh", "name": "GPT-5.2-2025-12-11 (xhigh reasoning)"}, {"id": "gpt-5.2-2025-12-11-no", "name": "GPT-5.2-2025-12-11 (no reasoning)"}, {"id": "gpt-5.4-2026-03-05-high", "name": "GPT-5.4-2026-03-05 (high reasoning)"}, {"id": "gpt-5.4-2026-03-05-low", "name": "GPT-5.4-2026-03-05 (low reasoning)"}, {"id": "gpt-5.4-2026-03-05-no", "name": "GPT-5.4-2026-03-05 (no reasoning)"}, {"id": "gpt-5.4-mini-2026-03-17-high", "name": "GPT-5.4-mini-2026-03-17 (high reasoning)"}, {"id": "gpt-5.4-mini-2026-03-17-no", "name": "GPT-5.4-mini-2026-03-17 (no reasoning)"}, {"id": "gpt-5.4-nano-2026-03-17-high", "name": "GPT-5.4-nano-2026-03-17 (high reasoning)"}, {"id": "gpt-5.4-nano-2026-03-17-no", "name": "GPT-5.4-nano-2026-03-17 (no reasoning)"}, {"id": "gemma-2-27b", "name": "Gemma-2-27b", "url": "https://huggingface.co/google/gemma-2-27b-it"}, {"id": "gemma-2-9b", "name": "Gemma-2-9b", "url": "https://huggingface.co/google/gemma-2-9b-it"}, {"id": "gemma-3-27b", "name": "Gemma-3-27b", "url": "https://huggingface.co/google/gemma-3-27b-it"}, {"id": "gemma-4-E2B-it", "name": "Gemma-4-E2B-it (reasoning disabled)", "url": "https://huggingface.co/google/gemma-4-E2B-it"}, {"id": "gemma-4-E2B", "name": "Gemma-4-E2B", "url": "https://huggingface.co/google/gemma-4-E2B"}, {"id": "gemma-4-E4B-it", "name": "Gemma-4-E4B-it (reasoning disabled)", "url": "https://huggingface.co/google/gemma-4-E4B-it"}, {"id": "gemma-4-E4B", "name": "Gemma-4-E4B", "url": "https://huggingface.co/google/gemma-4-E4B"}, {"id": "gemma-4-26B-A4B-it", "name": "Gemma-4-26B-A4B-it (reasoning disabled)", "url": "https://huggingface.co/google/gemma-4-26B-A4B-it"}, {"id": "gemma-4-26B-A4B", "name": "Gemma-4-26B-A4B", "url": "https://huggingface.co/google/gemma-4-26B-A4B"}, {"id": "gemma-4-31B-it", "name": "Gemma-4-31B-it (reasoning disabled)", "url": "https://huggingface.co/google/gemma-4-31B-it"}, {"id": "gemma-4-E2B-it-reasoning", "name": "Gemma-4-E2B-it (reasoning enabled)", "url": "https://huggingface.co/google/gemma-4-E2B-it"}, {"id": "gemma-4-E4B-it-reasoning", "name": "Gemma-4-E4B-it (reasoning enabled)", "url": "https://huggingface.co/google/gemma-4-E4B-it"}, {"id": "gemma-4-26B-A4B-it-reasoning", "name": "Gemma-4-26B-A4B-it (reasoning enabled)", "url": "https://huggingface.co/google/gemma-4-26B-A4B-it"}, {"id": "gemma-4-31B", "name": "Gemma-4-31B", "url": "https://huggingface.co/google/gemma-4-31B"}, {"id": "qwen-2.5-72b", "name": "Qwen-2.5-72b", "url": "https://huggingface.co/Qwen/Qwen2.5-72B-Instruct"}, {"id": "qwen-2.5-7b", "name": "Qwen-2.5-7b", "url": "https://huggingface.co/Qwen/Qwen2.5-7B-Instruct"}, {"id": "qwen-2.5-14B", "name": "Qwen-2.5-14b", "url": "https://huggingface.co/Qwen/Qwen2.5-14B-Instruct"}, {"id": "qwen-2.5-32B", "name": "Qwen-2.5-32b", "url": "https://huggingface.co/Qwen/Qwen2.5-32B-Instruct"}, {"id": "qwen-turbo-2024-11-01", "name": "Qwen-Turbo-2024-11-01"}, {"id": "qwen3.6-plus", "name": "Qwen3.6-Plus"}, {"id": "qwen-plus", "name": "Qwen-Plus"}, {"id": "qwen-max", "name": "Qwen-Max"}, {"id": "qwen3-max", "name": "Qwen3-Max"}, {"id": "qwen3.7-max", "name": "Qwen3.7-Max"}, {"id": "qwen3-235b-a22b", "name": "Qwen3-235B-A22B", "url": "https://huggingface.co/Qwen/Qwen3-235B-A22B"}, {"id": "qwen3-30b-a3b", "name": "Qwen3-30B-A3B", "url": "https://huggingface.co/Qwen/Qwen3-30B-A3B"}, {"id": "qwen3-32b", "name": "Qwen3-32B", "url": "https://huggingface.co/Qwen/Qwen3-32B"}, {"id": "qwen3-14b", "name": "Qwen3-14B", "url": "https://huggingface.co/Qwen/Qwen3-14B"}, {"id": "qwen3-8b", "name": "Qwen3-8B", "url": "https://huggingface.co/Qwen/Qwen3-8B"}, {"id": "qwen3-next-80b-a3b-instruct", "name": "Qwen3-Next-80B-A3B-Instruct", "url": "https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct"}, {"id": "qwen3-next-80b-a3b-thinking", "name": "Qwen3-Next-80B-A3B-Thinking", "url": "https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Thinking"}, {"id": "command-r-plus-04-2024", "name": "Command-R-Plus-04-2024", "url": "https://huggingface.co/CohereForAI/c4ai-command-r-plus"}, {"id": "command-r-plus-08-2024", "name": "Command-R-Plus-08-2024", "url": "https://huggingface.co/CohereForAI/c4ai-command-r-plus-08-2024"}, {"id": "command-r7b", "name": "Command-R7B", "url": "https://huggingface.co/CohereForAI/c4ai-command-r-plus-08-2024"}, {"id": "command-a-03-2025", "name": "Command-A-03-2025", "url": "https://huggingface.co/CohereForAI/c4ai-command-a-03-2025"}, {"id": "grok-2-1212", "name": "Grok-2-1212"}, {"id": "gemini-2.0-flash-experimental", "name": "Gemini-2.0-Flash-Experimental"}, {"id": "gemini-2.0-flash-thinking-exp-01-21", "name": "Gemini-2.0-Flash-Thinking-Exp-01-21"}, {"id": "gemini-2.5-pro-exp-03-25", "name": "Gemini-2.5-Pro-Exp-03-25"}, {"id": "gemini-2.5-flash-preview-04-17", "name": "Gemini-2.5-Flash-Preview-04-17"}, {"id": "gemini-2.5-pro-preview-06-05", "name": "Gemini-2.5-Pro-Preview-06-05"}, {"id": "gemini-3.0-pro-preview", "name": "Gemini-3.0-Pro-Preview"}, {"id": "gemini-3.1-pro-preview", "name": "Gemini-3.1-Pro-Preview"}, {"id": "gemini-3-flash-preview", "name": "Gemini-3-Flash-Preview"}, {"id": "gemini-3.5-flash", "name": "Gemini-3.5-Flash"}, {"id": "o1-mini-2024-09-12", "name": "O1-mini-2024-09-12"}, {"id": "o1-2024-12-17", "name": "O1-2024-12-17"}, {"id": "o3-mini-2025-01-31", "name": "O3-mini-2025-01-31"}, {"id": "o4-mini-2025-04-16", "name": "O4-Mini-2025-04-16"}, {"id": "o3-2025-04-16", "name": "O3-2025-04-16"}, {"id": "gemini-exp-1206", "name": "Gemini-Exp-1206"}, {"id": "deepseek-v4-flash", "name": "DeepSeek-V4-Flash", "url": "https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash"}, {"id": "deepseek-v4-pro", "name": "DeepSeek-V4-Pro", "url": "https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro"}, {"id": "deepseek-v3", "name": "DeepSeek-v3", "url": "https://huggingface.co/deepseek-ai/DeepSeek-V3"}, {"id": "deepseek-v3-0324", "name": "DeepSeek-v3-0324", "url": "https://huggingface.co/deepseek-ai/DeepSeek-V3-0324"}, {"id": "deepseek-v3.1", "name": "DeepSeek-v3.1 (no thinking)", "url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.1"}, {"id": "deepseek-v3.1-thinking", "name": "DeepSeek-v3.1 (thinking)", "url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.1"}, {"id": "deepseek-v3.2-exp", "name": "DeepSeek-v3.2-Exp", "url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.2-Exp"}, {"id": "deepseek-v3.2", "name": "DeepSeek-V3.2", "url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.2"}, {"id": "deepseek-v3.2-speciale", "name": "DeepSeek-V3.2-Speciale", "url": "https://huggingface.co/deepseek-ai/DeepSeek-V3.2-Speciale"}, {"id": "deepseek-r1", "name": "DeepSeek-R1", "url": "https://huggingface.co/deepseek-ai/DeepSeek-R1"}, {"id": "deepseek-r1-0528", "name": "DeepSeek-R1-0528", "url": "https://huggingface.co/deepseek-ai/DeepSeek-R1-0528"}, {"id": "gemini-pro-1.5", "name": "Gemini-Pro-1.5"}, {"id": "gemini-flash-1.5", "name": "Gemini-Flash-1.5"}, {"id": "phi-4", "name": "Phi-4", "url": "https://huggingface.co/microsoft/phi-4"}, {"id": "Phi-4-Multimodal-Instruct", "name": "Phi-4-Multimodal-Instruct", "url": "https://huggingface.co/microsoft/Phi-4-multimodal-instruct"}, {"id": "PLLuM-12B-nc-chat", "name": "PLLuM-12B-nc-chat", "url": "https://huggingface.co/CYFRAGOVPL/PLLuM-12B-nc-chat"}, {"id": "pllum-12b-nc-chat-250715", "name": "PLLuM-12B-nc-chat-250715", "url": "https://huggingface.co/CYFRAGOVPL/pllum-12b-nc-chat-250715"}, {"id": "PLLuM-8x7B-nc-chat", "name": "PLLuM-8x7B-nc-chat", "url": "https://huggingface.co/CYFRAGOVPL/PLLuM-8x7B-nc-chat"}, {"id": "llama-3.1-tulu-3-405b", "name": "Llama-3.1-Tulu-3-405B", "url": "https://huggingface.co/allenai/Llama-3.1-Tulu-3-405B"}, {"id": "Llama-PLLuM-8B-chat", "name": "Llama-PLLuM-8B-chat", "url": "https://huggingface.co/CYFRAGOVPL/Llama-PLLuM-8B-chat"}, {"id": "PLLuM-12B-chat", "name": "PLLuM-12B-chat", "url": "https://huggingface.co/CYFRAGOVPL/PLLuM-12B-chat"}, {"id": "Llama-PLLuM-70B-chat", "name": "Llama-PLLuM-70B-chat", "url": "https://huggingface.co/CYFRAGOVPL/Llama-PLLuM-70B-chat"}, {"id": "Llama-PLLuM-70B-chat-250801", "name": "Llama-PLLuM-70B-chat-250801", "url": "https://huggingface.co/CYFRAGOVPL/Llama-PLLuM-70B-chat-250801"}, {"id": "PLLuM-8x7B-chat", "name": "PLLuM-8x7B-chat", "url": "https://huggingface.co/CYFRAGOVPL/PLLuM-8x7B-chat"}, {"id": "Llama-PLLuM-8B-chat-2512", "name": "Llama-PLLuM-8B-chat-2512", "url": "https://huggingface.co/CYFRAGOVPL/Llama-PLLuM-8B-chat-2512"}, {"id": "PLLuM-4B-chat-2512", "name": "PLLuM-4B-chat-2512", "url": "https://huggingface.co/CYFRAGOVPL/PLLuM-4B-chat-2512"}, {"id": "PLLuM-12B-chat-2512", "name": "PLLuM-12B-chat-2512", "url": "https://huggingface.co/CYFRAGOVPL/PLLuM-12B-chat-2512"}, {"id": "Llama-PLLuM-70B-chat-2512", "name": "Llama-PLLuM-70B-chat-2512", "url": "https://huggingface.co/CYFRAGOVPL/Llama-PLLuM-70B-chat-2512"}, {"id": "EuroLLM-9B", "name": "EuroLLM-9B", "url": "https://huggingface.co/utter-project/EuroLLM-9B-Instruct"}, {"id": "Llama-4-Maverick", "name": "Llama-4-Maverick", "url": "https://huggingface.co/meta-llama/Llama-4-Maverick-17B-128E-Instruct"}, {"id": "Llama-4-Scout", "name": "Llama-4-Scout", "url": "https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E-Instruct"}, {"id": "grok-3", "name": "Grok-3-Beta"}, {"id": "grok-3-mini", "name": "Grok-3-Mini-Beta"}, {"id": "grok-4", "name": "Grok-4"}, {"id": "grok-4-fast", "name": "Grok-4-Fast"}, {"id": "grok-4.1-fast", "name": "Grok-4.1-Fast"}, {"id": "grok-4.20", "name": "Grok-4.20"}, {"id": "grok-4.3", "name": "Grok-4.3"}, {"id": "kimi-k2", "name": "Kimi-K2", "url": "https://huggingface.co/moonshotai/Kimi-K2-Instruct"}, {"id": "kimi-k2-0905", "name": "Kimi-K2-0905", "url": "https://huggingface.co/moonshotai/Kimi-K2-Instruct-0905"}, {"id": "kimi-k2-thinking", "name": "Kimi-K2-Thinking", "url": "https://huggingface.co/moonshotai/Kimi-K2-Thinking"}, {"id": "kimi-k2.5", "name": "Kimi-K2.5", "url": "https://huggingface.co/moonshotai/Kimi-K2.5"}, {"id": "kimi-k2.6", "name": "Kimi-K2.6", "url": "https://huggingface.co/moonshotai/Kimi-K2.6"}, {"id": "Kimi-K3", "name": "Kimi-K3", "url": "https://huggingface.co/moonshotai/Kimi-K3"}, {"id": "GLM-4.5", "name": "GLM-4.5", "url": "https://huggingface.co/zai-org/GLM-4.5"}, {"id": "GLM-4.6V", "name": "GLM-4.6V", "url": "https://huggingface.co/zai-org/GLM-4.6"}, {"id": "glm-4.7", "name": "GLM-4.7", "url": "https://huggingface.co/zai-org/GLM-4.7"}, {"id": "glm-5", "name": "GLM-5", "url": "https://huggingface.co/zai-org/GLM-5"}, {"id": "glm-5.1", "name": "GLM-5.1", "url": "https://huggingface.co/zai-org/GLM-5.1"}, {"id": "glm-4.7-flash", "name": "GLM-4.7-Flash", "url": "https://huggingface.co/zai-org/GLM-4.7-Flash"}, {"id": "glm-4.5-air", "name": "GLM-4.5-Air", "url": "https://huggingface.co/zai-org/GLM-4.5-Air"}, {"id": "gpt-oss-120b", "name": "GPT-OSS-120b", "url": "https://huggingface.co/openai/gpt-oss-120b"}, {"id": "gpt-oss-20b", "name": "GPT-OSS-20b", "url": "https://huggingface.co/openai/gpt-oss-20b"}, {"id": "minimax-m2.5", "name": "MiniMax-M2.5", "url": "https://huggingface.co/MiniMaxAI/MiniMax-M2.5"}, {"id": "minimax-m2.7", "name": "MiniMax-M2.7", "url": "https://huggingface.co/MiniMaxAI/MiniMax-M2.7"}, {"id": "Qwen3.5-397b-A17b", "name": "Qwen3.5-397B-A17B", "url": "https://huggingface.co/Qwen/Qwen3.5-397B-A17B"}, {"id": "Qwen3.5-27B", "name": "Qwen3.5-27B", "url": "https://huggingface.co/Qwen/Qwen3.5-27B"}, {"id": "Qwen3.6-27B", "name": "Qwen3.6-27B", "url": "https://huggingface.co/Qwen/Qwen3.6-27B"}, {"id": "Qwen3.5-35B-A3B", "name": "Qwen3.5-35B-A3B", "url": "https://huggingface.co/Qwen/Qwen3.5-35B-A3B"}, {"id": "Qwen3.6-35B-A3B", "name": "Qwen3.6-35B-A3B", "url": "https://huggingface.co/Qwen/Qwen3.6-35B-A3B"}, {"id": "Qwen3.5-122B-A10B", "name": "Qwen3.5-122B-A10B", "url": "https://huggingface.co/Qwen/Qwen3.5-122B-A10B"}, {"id": "Qwen3.5-9B", "name": "Qwen3.5-9B", "url": "https://huggingface.co/Qwen/Qwen3.5-9B"}, {"id": "Qwen3.5-4B", "name": "Qwen3.5-4B", "url": "https://huggingface.co/Qwen/Qwen3.5-4B"}, {"id": "Qwen3.5-2B", "name": "Qwen3.5-2B", "url": "https://huggingface.co/Qwen/Qwen3.5-2B"}, {"id": "mimo-v2-pro", "name": "MiMo-V2-Pro"}, {"id": "MiMo-V2-Omni", "name": "MiMo-V2-Omni"}, {"id": "MiMo-V2.5", "name": "MiMo-V2.5", "url": "https://huggingface.co/XiaomiMiMo/MiMo-V2.5"}, {"id": "InternVL3.5-38B", "name": "InternVL3.5-38B", "url": "https://huggingface.co/OpenGVLab/InternVL3_5-38B"}, {"id": "InternVL3.5-30B-A3B", "name": "InternVL3.5-30B-A3B", "url": "https://huggingface.co/OpenGVLab/InternVL3_5-30B-A3B"}, {"id": "InternVL3.5-14B", "name": "InternVL3.5-14B", "url": "https://huggingface.co/OpenGVLab/InternVL3_5-14B"}, {"id": "InternVL3.5-8B", "name": "InternVL3.5-8B", "url": "https://huggingface.co/OpenGVLab/InternVL3_5-8B"}, {"id": "LLaVA-Bielik-11b-v2.6-instruct", "name": "LLaVA-Bielik-11b-v2.6-instruct", "url": "https://huggingface.co/NASK-PIB/LLaVA-Bielik-11b-v2.6-instruct"}, {"id": "LLaVA-PLLuM-12b-nc-instruct", "name": "LLaVA-PLLuM-12b-nc-instruct", "url": "https://huggingface.co/NASK-PIB/LLaVA-PLLuM-12b-nc-instruct"}, {"id": "Inkling", "name": "Inkling", "url": "https://huggingface.co/thinkingmachines/Inkling"}, {"id": "Inkling-Small", "name": "Inkling-Small", "url": "https://huggingface.co/thinkingmachines/Inkling-Small"}, {"id": "Muse-Glimmer-30b", "name": "Muse-Glimmer-30b", "url": "https://huggingface.co/meta-models/Muse-Glimmer-30B"}], "results": [{"id": "Gemini-3-Flash", "tags": [], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 70.0, "Soft": 76.25}, "Contemporary life": {"Strict": 74.0, "Soft": 80.2}, "Geography and environment": {"Strict": 69.0, "Soft": 79.69}, "ASR": {"Strict": 75.0, "Soft": 88.05}, "Speech QA": {"Strict": 69.0, "Soft": 75.68}, "Sound and music QA": {"Strict": 35.0, "Soft": 40.88}, "OCR": {"Strict": 84.0, "Soft": 92.5}, "Document QA": {"Strict": 64.0, "Soft": 76.84}, "Structured extraction": {"Strict": 68.0, "Soft": 96.65}}, "columns": {"Provider": "Google"}}, {"id": "Gemini-3.1-Pro", "tags": [], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 82.0, "Soft": 86.92}, "Contemporary life": {"Strict": 83.0, "Soft": 87.28}, "Geography and environment": {"Strict": 85.0, "Soft": 91.08}, "ASR": {"Strict": 77.0, "Soft": 91.03}, "Speech QA": {"Strict": 91.0, "Soft": 93.47}, "Sound and music QA": {"Strict": 56.0, "Soft": 59.67}, "OCR": {"Strict": 81.0, "Soft": 91.38}, "Document QA": {"Strict": 83.0, "Soft": 89.67}, "Structured extraction": {"Strict": 79.0, "Soft": 97.35}}, "columns": {"Provider": "Google"}}, {"id": "Gemini-3.1-Flash-Lite", "tags": [], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 58.0, "Soft": 63.17}, "Contemporary life": {"Strict": 49.0, "Soft": 58.16}, "Geography and environment": {"Strict": 62.0, "Soft": 76.15}, "ASR": {"Strict": 70.0, "Soft": 87.13}, "Speech QA": {"Strict": 35.0, "Soft": 47.16}, "Sound and music QA": {"Strict": 31.0, "Soft": 38.58}, "OCR": {"Strict": 76.0, "Soft": 91.06}, "Document QA": {"Strict": 44.0, "Soft": 60.0}, "Structured extraction": {"Strict": 45.0, "Soft": 91.05}}, "columns": {"Provider": "Google"}}, {"id": "GPT-5.4 + GPT-Audio", "tags": ["main"], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 47.0, "Soft": 56.08}, "Contemporary life": {"Strict": 42.0, "Soft": 52.75}, "Geography and environment": {"Strict": 55.0, "Soft": 64.34}, "ASR": {"Strict": 66.0, "Soft": 83.93}, "Speech QA": {"Strict": 41.0, "Soft": 53.1}, "Sound and music QA": {"Strict": 29.0, "Soft": 36.55}, "OCR": {"Strict": 71.0, "Soft": 89.72}, "Document QA": {"Strict": 54.0, "Soft": 67.21}, "Structured extraction": {"Strict": 45.0, "Soft": 91.41}}, "columns": {"Provider": "OpenAI"}}, {"id": "GPT-5.5 + GPT-Audio", "tags": ["main"], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 74.0, "Soft": 79.15}, "Contemporary life": {"Strict": 73.0, "Soft": 78.98}, "Geography and environment": {"Strict": 77.0, "Soft": 84.5}, "ASR": {"Strict": 66.0, "Soft": 83.93}, "Speech QA": {"Strict": 41.0, "Soft": 53.1}, "Sound and music QA": {"Strict": 29.0, "Soft": 36.55}, "OCR": {"Strict": 77.0, "Soft": 92.32}, "Document QA": {"Strict": 83.0, "Soft": 91.35}, "Structured extraction": {"Strict": 55.0, "Soft": 94.5}}, "columns": {"Provider": "OpenAI"}}, {"id": "GPT-5-Mini + GPT-Audio-Mini", "tags": ["main"], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 47.0, "Soft": 52.12}, "Contemporary life": {"Strict": 35.0, "Soft": 40.51}, "Geography and environment": {"Strict": 50.0, "Soft": 61.2}, "ASR": {"Strict": 62.0, "Soft": 82.55}, "Speech QA": {"Strict": 23.0, "Soft": 36.8}, "Sound and music QA": {"Strict": 16.0, "Soft": 22.17}, "OCR": {"Strict": 60.0, "Soft": 84.74}, "Document QA": {"Strict": 62.0, "Soft": 72.67}, "Structured extraction": {"Strict": 36.0, "Soft": 85.48}}, "columns": {"Provider": "OpenAI"}}, {"id": "GPT-4o + GPT-4o-Audio", "tags": ["main"], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 41.0, "Soft": 50.7}, "Contemporary life": {"Strict": 39.0, "Soft": 47.25}, "Geography and environment": {"Strict": 48.0, "Soft": 58.87}, "ASR": {"Strict": 64.0, "Soft": 82.57}, "Speech QA": {"Strict": 50.0, "Soft": 59.33}, "Sound and music QA": {"Strict": 34.0, "Soft": 40.0}, "OCR": {"Strict": 17.0, "Soft": 31.78}, "Document QA": {"Strict": 23.0, "Soft": 44.38}, "Structured extraction": {"Strict": 25.0, "Soft": 80.23}}, "columns": {"Provider": "OpenAI"}}, {"id": "MiMo-V2-Omni", "tags": [], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 21.0, "Soft": 30.6}, "Contemporary life": {"Strict": 27.0, "Soft": 33.55}, "Geography and environment": {"Strict": 28.0, "Soft": 42.87}, "ASR": {"Strict": 3.0, "Soft": 48.56}, "Speech QA": {"Strict": 14.0, "Soft": 28.48}, "Sound and music QA": {"Strict": 23.0, "Soft": 28.0}, "OCR": {"Strict": 57.0, "Soft": 82.23}, "Document QA": {"Strict": 53.0, "Soft": 68.06}, "Structured extraction": {"Strict": 16.0, "Soft": 39.51}}, "columns": {"Provider": "Xiaomi"}}, {"id": "MiMo-V2.5", "tags": [], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 22.0, "Soft": 30.57}, "Contemporary life": {"Strict": 23.0, "Soft": 32.51}, "Geography and environment": {"Strict": 31.0, "Soft": 42.04}, "ASR": {"Strict": 1.0, "Soft": 49.12}, "Speech QA": {"Strict": 15.0, "Soft": 29.4}, "Sound and music QA": {"Strict": 22.0, "Soft": 28.4}, "OCR": {"Strict": 57.0, "Soft": 82.75}, "Document QA": {"Strict": 62.0, "Soft": 72.1}, "Structured extraction": {"Strict": 39.0, "Soft": 88.65}}, "columns": {"Provider": "Xiaomi"}}, {"id": "Gemini-2.5-Pro", "tags": [], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 70.0, "Soft": 76.0}, "Contemporary life": {"Strict": 72.0, "Soft": 80.16}, "Geography and environment": {"Strict": 72.0, "Soft": 80.8}, "ASR": {"Strict": 81.0, "Soft": 91.45}, "Speech QA": {"Strict": 65.0, "Soft": 76.07}, "Sound and music QA": {"Strict": 45.0, "Soft": 52.97}, "OCR": {"Strict": 80.0, "Soft": 92.29}, "Document QA": {"Strict": 72.0, "Soft": 83.6}, "Structured extraction": {"Strict": 68.0, "Soft": 96.44}}, "columns": {"Provider": "Google"}}, {"id": "Gemini-2.5-Flash", "tags": [], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 50.0, "Soft": 58.18}, "Contemporary life": {"Strict": 40.0, "Soft": 51.04}, "Geography and environment": {"Strict": 53.0, "Soft": 66.16}, "ASR": {"Strict": 69.0, "Soft": 89.71}, "Speech QA": {"Strict": 41.0, "Soft": 52.2}, "Sound and music QA": {"Strict": 23.0, "Soft": 29.78}, "OCR": {"Strict": 73.0, "Soft": 91.01}, "Document QA": {"Strict": 52.0, "Soft": 64.42}, "Structured extraction": {"Strict": 53.0, "Soft": 91.86}}, "columns": {"Provider": "Google"}}, {"id": "Phi-4-Multimodal-Instruct", "tags": [], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 2.0, "Soft": 6.17}, "Contemporary life": {"Strict": 2.0, "Soft": 3.42}, "Geography and environment": {"Strict": 5.0, "Soft": 11.83}, "ASR": {"Strict": 0.0, "Soft": 2.21}, "Speech QA": {"Strict": 0.0, "Soft": 2.16}, "Sound and music QA": {"Strict": 2.0, "Soft": 4.01}, "OCR": {"Strict": 0.0, "Soft": 0.57}, "Document QA": {"Strict": 6.0, "Soft": 11.8}, "Structured extraction": {"Strict": 0.0, "Soft": 14.85}}, "columns": {"Provider": "Microsoft"}}, {"id": "Mistral-3.2-24B + Voxtral-24B", "tags": ["main"], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 15.0, "Soft": 21.95}, "Contemporary life": {"Strict": 17.0, "Soft": 24.87}, "Geography and environment": {"Strict": 7.0, "Soft": 19.94}, "ASR": {"Strict": 36.0, "Soft": 80.27}, "Speech QA": {"Strict": 23.0, "Soft": 35.43}, "Sound and music QA": {"Strict": 4.0, "Soft": 12.51}, "OCR": {"Strict": 1.0, "Soft": 1.14}, "Document QA": {"Strict": 18.0, "Soft": 36.49}, "Structured extraction": {"Strict": 18.0, "Soft": 78.85}}, "columns": {"Provider": "Mistral"}}, {"id": "Qwen3.5-397B-A17B", "tags": [], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 51.0, "Soft": 56.49}, "Contemporary life": {"Strict": 40.0, "Soft": 49.94}, "Geography and environment": {"Strict": 51.0, "Soft": 63.27}, "OCR": {"Strict": 78.0, "Soft": 92.52}, "Document QA": {"Strict": 78.0, "Soft": 87.77}, "Structured extraction": {"Strict": 35.0, "Soft": 89.46}}, "columns": {"Provider": "Alibaba"}}, {"id": "Qwen3.5-122B-A10B", "tags": [], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 35.0, "Soft": 44.27}, "Contemporary life": {"Strict": 28.0, "Soft": 36.35}, "Geography and environment": {"Strict": 29.0, "Soft": 44.18}, "OCR": {"Strict": 73.0, "Soft": 90.18}, "Document QA": {"Strict": 64.0, "Soft": 76.0}, "Structured extraction": {"Strict": 36.0, "Soft": 88.14}}, "columns": {"Provider": "Alibaba"}}, {"id": "Qwen3.6-35B-A3B", "tags": [], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 23.0, "Soft": 31.09}, "Contemporary life": {"Strict": 21.0, "Soft": 27.79}, "Geography and environment": {"Strict": 22.0, "Soft": 35.85}, "OCR": {"Strict": 69.0, "Soft": 89.29}, "Document QA": {"Strict": 60.0, "Soft": 72.3}, "Structured extraction": {"Strict": 36.0, "Soft": 89.21}}, "columns": {"Provider": "Alibaba"}}, {"id": "Qwen3.5-35B-A3B", "tags": [], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 23.0, "Soft": 31.3}, "Contemporary life": {"Strict": 20.0, "Soft": 26.1}, "Geography and environment": {"Strict": 22.0, "Soft": 33.0}, "OCR": {"Strict": 67.0, "Soft": 85.94}, "Document QA": {"Strict": 61.0, "Soft": 72.82}, "Structured extraction": {"Strict": 33.0, "Soft": 68.01}}, "columns": {"Provider": "Alibaba"}}, {"id": "Qwen3.6-27B", "tags": [], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 22.0, "Soft": 31.06}, "Contemporary life": {"Strict": 19.0, "Soft": 25.68}, "Geography and environment": {"Strict": 19.0, "Soft": 32.74}, "OCR": {"Strict": 61.0, "Soft": 81.49}, "Document QA": {"Strict": 62.0, "Soft": 78.02}, "Structured extraction": {"Strict": 36.0, "Soft": 70.67}}, "columns": {"Provider": "Alibaba"}}, {"id": "Qwen3.5-27B", "tags": [], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 24.0, "Soft": 31.1}, "Contemporary life": {"Strict": 16.0, "Soft": 19.11}, "Geography and environment": {"Strict": 18.0, "Soft": 30.05}, "OCR": {"Strict": 66.0, "Soft": 85.87}, "Document QA": {"Strict": 63.0, "Soft": 72.47}, "Structured extraction": {"Strict": 37.0, "Soft": 88.96}}, "columns": {"Provider": "Alibaba"}}, {"id": "Qwen3.5-9B", "tags": [], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 10.0, "Soft": 14.97}, "Contemporary life": {"Strict": 10.0, "Soft": 16.93}, "Geography and environment": {"Strict": 8.0, "Soft": 17.1}, "OCR": {"Strict": 55.0, "Soft": 76.9}, "Document QA": {"Strict": 50.0, "Soft": 59.4}, "Structured extraction": {"Strict": 30.0, "Soft": 66.38}}, "columns": {"Provider": "Alibaba"}}, {"id": "Qwen3.5-4B", "tags": [], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 11.0, "Soft": 16.93}, "Contemporary life": {"Strict": 6.0, "Soft": 12.84}, "Geography and environment": {"Strict": 7.0, "Soft": 16.96}, "OCR": {"Strict": 42.0, "Soft": 78.44}, "Document QA": {"Strict": 39.0, "Soft": 54.28}, "Structured extraction": {"Strict": 23.0, "Soft": 82.28}}, "columns": {"Provider": "Alibaba"}}, {"id": "Qwen3.5-2B", "tags": [], "timestamp": "08/05/2026,15:54:27", "results": {"History and culture": {"Strict": 2.0, "Soft": 6.49}, "Contemporary life": {"Strict": 4.0, "Soft": 9.49}, "Geography and environment": {"Strict": 2.0, "Soft": 10.92}, "OCR": {"Strict": 28.0, "Soft": 69.18}, "Document QA": {"Strict": 5.0, "Soft": 18.47}, "Structured extraction": {"Strict": 12.0, "Soft": 65.58}}, "columns": {"Provider": "Alibaba"}}, {"id": "Gemma-4-31B", "tags": [], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 36.0, "Soft": 45.74}, "Contemporary life": {"Strict": 32.0, "Soft": 42.87}, "Geography and environment": {"Strict": 47.0, "Soft": 57.7}, "OCR": {"Strict": 76.0, "Soft": 92.34}, "Document QA": {"Strict": 70.0, "Soft": 82.3}, "Structured extraction": {"Strict": 59.0, "Soft": 94.95}}, "columns": {"Provider": "Google"}}, {"id": "Gemma-4-26B-A4B", "tags": [], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 39.0, "Soft": 48.14}, "Contemporary life": {"Strict": 26.0, "Soft": 35.38}, "Geography and environment": {"Strict": 32.0, "Soft": 46.59}, "OCR": {"Strict": 62.0, "Soft": 87.26}, "Document QA": {"Strict": 57.0, "Soft": 72.21}, "Structured extraction": {"Strict": 40.0, "Soft": 89.68}}, "columns": {"Provider": "Google"}}, {"id": "Gemma-4-E4B", "tags": [], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 9.0, "Soft": 15.81}, "Contemporary life": {"Strict": 10.0, "Soft": 14.63}, "Geography and environment": {"Strict": 8.0, "Soft": 17.65}, "OCR": {"Strict": 42.0, "Soft": 80.39}, "Document QA": {"Strict": 35.0, "Soft": 51.3}, "Structured extraction": {"Strict": 21.0, "Soft": 77.8}}, "columns": {"Provider": "Google"}}, {"id": "Gemma-4-E2B", "tags": [], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 4.0, "Soft": 6.67}, "Contemporary life": {"Strict": 4.0, "Soft": 9.38}, "Geography and environment": {"Strict": 4.0, "Soft": 14.11}, "OCR": {"Strict": 30.0, "Soft": 72.31}, "Document QA": {"Strict": 21.0, "Soft": 34.23}, "Structured extraction": {"Strict": 15.0, "Soft": 73.25}}, "columns": {"Provider": "Google"}}, {"id": "Grok-4", "tags": [], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 53.0, "Soft": 62.43}, "Contemporary life": {"Strict": 47.0, "Soft": 57.2}, "Geography and environment": {"Strict": 55.0, "Soft": 67.8}, "OCR": {"Strict": 17.0, "Soft": 59.54}, "Document QA": {"Strict": 47.0, "Soft": 64.26}, "Structured extraction": {"Strict": 19.0, "Soft": 76.87}}, "columns": {"Provider": "xAI"}}, {"id": "Grok-4.1-Fast", "tags": [], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 36.0, "Soft": 45.63}, "Contemporary life": {"Strict": 32.0, "Soft": 43.29}, "Geography and environment": {"Strict": 39.0, "Soft": 50.66}, "OCR": {"Strict": 1.0, "Soft": 30.55}, "Document QA": {"Strict": 28.0, "Soft": 44.41}, "Structured extraction": {"Strict": 6.0, "Soft": 54.81}}, "columns": {"Provider": "xAI"}}, {"id": "Grok-4.20", "tags": [], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 36.0, "Soft": 45.32}, "Contemporary life": {"Strict": 25.0, "Soft": 34.03}, "Geography and environment": {"Strict": 39.0, "Soft": 50.1}, "OCR": {"Strict": 21.0, "Soft": 31.99}, "Document QA": {"Strict": 19.0, "Soft": 39.63}, "Structured extraction": {"Strict": 34.0, "Soft": 87.42}}, "columns": {"Provider": "xAI"}}, {"id": "GLM-5V-Turbo", "tags": [], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 45.0, "Soft": 53.97}, "Contemporary life": {"Strict": 35.0, "Soft": 47.32}, "Geography and environment": {"Strict": 42.0, "Soft": 55.83}, "OCR": {"Strict": 34.0, "Soft": 73.54}, "Document QA": {"Strict": 48.0, "Soft": 64.19}, "Structured extraction": {"Strict": 23.0, "Soft": 81.32}}, "columns": {"Provider": "Zhipu AI"}}, {"id": "GLM-4.6V", "tags": [], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 31.0, "Soft": 38.22}, "Contemporary life": {"Strict": 20.0, "Soft": 30.24}, "Geography and environment": {"Strict": 23.0, "Soft": 36.5}, "OCR": {"Strict": 36.0, "Soft": 73.84}, "Document QA": {"Strict": 41.0, "Soft": 57.69}, "Structured extraction": {"Strict": 12.0, "Soft": 51.74}}, "columns": {"Provider": "Zhipu AI"}}, {"id": "InternVL3.5-38B", "tags": [], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 5.0, "Soft": 10.0}, "Contemporary life": {"Strict": 7.0, "Soft": 13.96}, "Geography and environment": {"Strict": 11.0, "Soft": 20.11}, "OCR": {"Strict": 5.0, "Soft": 22.75}, "Document QA": {"Strict": 20.0, "Soft": 37.54}, "Structured extraction": {"Strict": 12.0, "Soft": 69.92}}, "columns": {"Provider": "Shanghai AI"}}, {"id": "InternVL3.5-30B-A3B", "tags": [], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 3.0, "Soft": 6.38}, "Contemporary life": {"Strict": 8.0, "Soft": 12.79}, "Geography and environment": {"Strict": 4.0, "Soft": 11.74}, "OCR": {"Strict": 2.0, "Soft": 36.17}, "Document QA": {"Strict": 8.0, "Soft": 22.83}, "Structured extraction": {"Strict": 8.0, "Soft": 61.88}}, "columns": {"Provider": "Shanghai AI"}}, {"id": "InternVL3.5-14B", "tags": [], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 1.0, "Soft": 7.12}, "Contemporary life": {"Strict": 7.0, "Soft": 13.81}, "Geography and environment": {"Strict": 5.0, "Soft": 13.05}, "OCR": {"Strict": 4.0, "Soft": 30.32}, "Document QA": {"Strict": 17.0, "Soft": 30.17}, "Structured extraction": {"Strict": 5.0, "Soft": 63.4}}, "columns": {"Provider": "Shanghai AI"}}, {"id": "InternVL3.5-8B", "tags": [], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 2.0, "Soft": 6.73}, "Contemporary life": {"Strict": 6.0, "Soft": 11.69}, "Geography and environment": {"Strict": 4.0, "Soft": 10.45}, "OCR": {"Strict": 1.0, "Soft": 18.86}, "Document QA": {"Strict": 9.0, "Soft": 22.46}, "Structured extraction": {"Strict": 9.0, "Soft": 58.61}}, "columns": {"Provider": "Shanghai AI"}}, {"id": "Claude-Opus-4.7", "tags": [], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 62.0, "Soft": 69.86}, "Contemporary life": {"Strict": 36.0, "Soft": 44.43}, "Geography and environment": {"Strict": 62.0, "Soft": 73.97}, "OCR": {"Strict": 88.0, "Soft": 94.9}, "Document QA": {"Strict": 63.0, "Soft": 73.86}, "Structured extraction": {"Strict": 59.0, "Soft": 94.59}}, "columns": {"Provider": "Anthropic"}}, {"id": "Claude-Sonnet-4.6", "tags": [], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 54.0, "Soft": 58.83}, "Contemporary life": {"Strict": 26.0, "Soft": 34.91}, "Geography and environment": {"Strict": 53.0, "Soft": 64.01}, "OCR": {"Strict": 79.0, "Soft": 92.51}, "Document QA": {"Strict": 49.0, "Soft": 66.03}, "Structured extraction": {"Strict": 52.0, "Soft": 92.85}}, "columns": {"Provider": "Anthropic"}}, {"id": "Claude-Haiku-4.5", "tags": [], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 20.0, "Soft": 28.47}, "Contemporary life": {"Strict": 9.0, "Soft": 15.38}, "Geography and environment": {"Strict": 21.0, "Soft": 30.79}, "OCR": {"Strict": 22.0, "Soft": 61.55}, "Document QA": {"Strict": 11.0, "Soft": 29.16}, "Structured extraction": {"Strict": 19.0, "Soft": 73.41}}, "columns": {"Provider": "Anthropic"}}, {"id": "Mistral-Large-3-2512", "tags": [], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 24.0, "Soft": 32.08}, "Contemporary life": {"Strict": 18.0, "Soft": 24.37}, "Geography and environment": {"Strict": 22.0, "Soft": 34.75}, "OCR": {"Strict": 5.0, "Soft": 7.86}, "Document QA": {"Strict": 19.0, "Soft": 38.13}, "Structured extraction": {"Strict": 15.0, "Soft": 78.01}}, "columns": {"Provider": "Mistral"}}, {"id": "Mistral-Medium-3.1", "tags": [], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 36.0, "Soft": 44.44}, "Contemporary life": {"Strict": 20.0, "Soft": 29.36}, "Geography and environment": {"Strict": 27.0, "Soft": 40.36}, "OCR": {"Strict": 6.0, "Soft": 9.81}, "Document QA": {"Strict": 14.0, "Soft": 34.8}, "Structured extraction": {"Strict": 16.0, "Soft": 79.0}}, "columns": {"Provider": "Mistral"}}, {"id": "Mistral-Small-4", "tags": [], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 13.0, "Soft": 19.88}, "Contemporary life": {"Strict": 14.0, "Soft": 18.72}, "Geography and environment": {"Strict": 13.0, "Soft": 21.33}, "OCR": {"Strict": 13.0, "Soft": 28.37}, "Document QA": {"Strict": 21.0, "Soft": 38.47}, "Structured extraction": {"Strict": 18.0, "Soft": 70.94}}, "columns": {"Provider": "Mistral"}}, {"id": "Mistral-3.2-24B", "tags": [], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 15.0, "Soft": 21.95}, "Contemporary life": {"Strict": 17.0, "Soft": 24.87}, "Geography and environment": {"Strict": 7.0, "Soft": 19.94}, "OCR": {"Strict": 1.0, "Soft": 1.14}, "Document QA": {"Strict": 18.0, "Soft": 36.49}, "Structured extraction": {"Strict": 18.0, "Soft": 78.85}}, "columns": {"Provider": "Mistral"}}, {"id": "GPT-5.4", "tags": [], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 47.0, "Soft": 56.08}, "Contemporary life": {"Strict": 42.0, "Soft": 52.75}, "Geography and environment": {"Strict": 55.0, "Soft": 64.34}, "OCR": {"Strict": 71.0, "Soft": 89.72}, "Document QA": {"Strict": 54.0, "Soft": 67.21}, "Structured extraction": {"Strict": 45.0, "Soft": 91.41}}, "columns": {"Provider": "OpenAI"}}, {"id": "GPT-5.5", "tags": [], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 74.0, "Soft": 79.15}, "Contemporary life": {"Strict": 73.0, "Soft": 78.98}, "Geography and environment": {"Strict": 77.0, "Soft": 84.5}, "OCR": {"Strict": 77.0, "Soft": 92.32}, "Document QA": {"Strict": 83.0, "Soft": 91.35}, "Structured extraction": {"Strict": 55.0, "Soft": 94.5}}, "columns": {"Provider": "OpenAI"}}, {"id": "GPT-5.4-Mini", "tags": [], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 47.0, "Soft": 52.12}, "Contemporary life": {"Strict": 35.0, "Soft": 40.51}, "Geography and environment": {"Strict": 50.0, "Soft": 61.2}, "OCR": {"Strict": 60.0, "Soft": 84.74}, "Document QA": {"Strict": 62.0, "Soft": 72.67}, "Structured extraction": {"Strict": 36.0, "Soft": 85.48}}, "columns": {"Provider": "OpenAI"}}, {"id": "GPT-5.4-Nano", "tags": [], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 20.0, "Soft": 27.59}, "Contemporary life": {"Strict": 20.0, "Soft": 25.56}, "Geography and environment": {"Strict": 22.0, "Soft": 32.95}, "OCR": {"Strict": 17.0, "Soft": 43.67}, "Document QA": {"Strict": 28.0, "Soft": 43.84}, "Structured extraction": {"Strict": 11.0, "Soft": 59.53}}, "columns": {"Provider": "OpenAI"}}, {"id": "GPT-4o", "tags": [], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 41.0, "Soft": 50.7}, "Contemporary life": {"Strict": 39.0, "Soft": 47.25}, "Geography and environment": {"Strict": 48.0, "Soft": 58.87}, "OCR": {"Strict": 17.0, "Soft": 31.78}, "Document QA": {"Strict": 23.0, "Soft": 44.38}, "Structured extraction": {"Strict": 25.0, "Soft": 80.23}}, "columns": {"Provider": "OpenAI"}}, {"id": "Kimi-K2.6", "tags": [], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 50.0, "Soft": 59.88}, "Contemporary life": {"Strict": 40.0, "Soft": 52.56}, "Geography and environment": {"Strict": 47.0, "Soft": 60.61}, "OCR": {"Strict": 67.0, "Soft": 89.6}, "Document QA": {"Strict": 71.0, "Soft": 82.7}, "Structured extraction": {"Strict": 46.0, "Soft": 91.25}}, "columns": {"Provider": "Moonshot.AI"}}, {"id": "Kimi-K2.5", "tags": [], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 58.0, "Soft": 65.31}, "Contemporary life": {"Strict": 44.0, "Soft": 56.45}, "Geography and environment": {"Strict": 52.0, "Soft": 64.78}, "OCR": {"Strict": 73.0, "Soft": 90.11}, "Document QA": {"Strict": 65.0, "Soft": 80.03}, "Structured extraction": {"Strict": 45.0, "Soft": 92.26}}, "columns": {"Provider": "Moonshot.AI"}}, {"id": "Llama-4-Maverick", "tags": [], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 21.0, "Soft": 29.81}, "Contemporary life": {"Strict": 19.0, "Soft": 28.38}, "Geography and environment": {"Strict": 26.0, "Soft": 38.54}, "OCR": {"Strict": 28.0, "Soft": 68.09}, "Document QA": {"Strict": 19.0, "Soft": 38.32}, "Structured extraction": {"Strict": 19.0, "Soft": 78.94}}, "columns": {"Provider": "Meta"}}, {"id": "Llama-4-Scout", "tags": [], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 13.0, "Soft": 19.65}, "Contemporary life": {"Strict": 13.0, "Soft": 19.89}, "Geography and environment": {"Strict": 17.0, "Soft": 30.66}, "OCR": {"Strict": 23.0, "Soft": 64.99}, "Document QA": {"Strict": 21.0, "Soft": 38.72}, "Structured extraction": {"Strict": 18.0, "Soft": 76.41}}, "columns": {"Provider": "Meta"}}, {"id": "LLaVA-Bielik-11b-v2.6-instruct", "tags": [], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 11.0, "Soft": 19.66}, "Contemporary life": {"Strict": 15.0, "Soft": 22.69}, "Geography and environment": {"Strict": 11.0, "Soft": 20.92}, "OCR": {"Strict": 2.0, "Soft": 29.7}, "Document QA": {"Strict": 3.0, "Soft": 13.42}, "Structured extraction": {"Strict": 0.0, "Soft": 24.99}}, "columns": {"Provider": "NASK"}}, {"id": "LLaVA-PLLuM-12b-nc-instruct", "tags": [], "timestamp": "03/05/2026,18:07:41", "results": {"History and culture": {"Strict": 18.0, "Soft": 24.51}, "Contemporary life": {"Strict": 18.0, "Soft": 24.51}, "Geography and environment": {"Strict": 12.0, "Soft": 18.32}, "OCR": {"Strict": 3.0, "Soft": 25.4}, "Document QA": {"Strict": 5.0, "Soft": 14.45}, "Structured extraction": {"Strict": 2.0, "Soft": 24.16}}, "columns": {"Provider": "NASK"}}, {"id": "Grok-4.3", "tags": [], "timestamp": "05/05/2026,08:31:22", "results": {"History and culture": {"Strict": 55.0, "Soft": 62.39}, "Contemporary life": {"Strict": 42.0, "Soft": 50.4}, "Geography and environment": {"Strict": 52.0, "Soft": 64.68}, "OCR": {"Strict": 66.0, "Soft": 85.91}, "Document QA": {"Strict": 57.0, "Soft": 71.0}, "Structured extraction": {"Strict": 39.0, "Soft": 88.65}}, "columns": {"Provider": "xAI"}}, {"id": "Mistral-Medium-3.5", "tags": [], "timestamp": "05/05/2026,08:31:22", "results": {"History and culture": {"Strict": 33.0, "Soft": 40.24}, "Contemporary life": {"Strict": 29.0, "Soft": 38.71}, "Geography and environment": {"Strict": 31.0, "Soft": 42.16}, "OCR": {"Strict": 40.0, "Soft": 74.81}, "Document QA": {"Strict": 36.0, "Soft": 57.22}, "Structured extraction": {"Strict": 27.0, "Soft": 80.05}}, "columns": {"Provider": "Mistral"}}, {"id": "Gemini-3.5-Flash", "tags": [], "timestamp": "20/05/2026,09:30:45", "results": {"History and culture": {"Strict": 82.0, "Soft": 85.76}, "Contemporary life": {"Strict": 79.0, "Soft": 85.3}, "Geography and environment": {"Strict": 86.0, "Soft": 92.58}, "ASR": {"Strict": 77.0, "Soft": 89.22}, "Speech QA": {"Strict": 81.0, "Soft": 86.3}, "Sound and music QA": {"Strict": 47.0, "Soft": 51.5}, "OCR": {"Strict": 86.0, "Soft": 94.29}, "Document QA": {"Strict": 88.0, "Soft": 94.71}, "Structured extraction": {"Strict": 66.0, "Soft": 94.89}}, "columns": {"Provider": "Google"}}, {"id": "Claude-Fable-5", "tags": [], "timestamp": "09/06/2026,20:50:48", "results": {"History and culture": {"Strict": 76.0, "Soft": 79.71}, "Contemporary life": {"Strict": 62.0, "Soft": 70.13}, "Geography and environment": {"Strict": 85.0, "Soft": 89.83}, "OCR": {"Strict": 90.0, "Soft": 94.24}, "Document QA": {"Strict": 85.0, "Soft": 91.15}, "Structured extraction": {"Strict": 70.0, "Soft": 95.95}}, "columns": {"Provider": "Anthropic"}}, {"id": "Claude-Sonnet-5", "tags": [], "timestamp": "01/07/2026,08:08:15", "results": {"History and culture": {"Strict": 49.0, "Soft": 55.37}, "Contemporary life": {"Strict": 32.0, "Soft": 39.73}, "Geography and environment": {"Strict": 52.0, "Soft": 63.3}, "OCR": {"Strict": 79.0, "Soft": 91.79}, "Document QA": {"Strict": 68.0, "Soft": 80.03}, "Structured extraction": {"Strict": 60.0, "Soft": 95.09}}, "columns": {"Provider": "Anthropic"}}, {"id": "GPT-5.6-Sol", "tags": [], "timestamp": "17/07/2026,14:18:34", "results": {"History and culture": {"Strict": 73.0, "Soft": 76.08}, "Contemporary life": {"Strict": 75.0, "Soft": 81.71}, "Geography and environment": {"Strict": 78.0, "Soft": 85.83}, "OCR": {"Strict": 74.0, "Soft": 89.69}, "Document QA": {"Strict": 80.0, "Soft": 90.2}, "Structured extraction": {"Strict": 50.0, "Soft": 92.36}}, "columns": {"Provider": "OpenAI"}}, {"id": "GPT-5.6-Terra", "tags": [], "timestamp": "17/07/2026,14:32:30", "results": {"History and culture": {"Strict": 62.0, "Soft": 67.52}, "Contemporary life": {"Strict": 49.0, "Soft": 59.77}, "Geography and environment": {"Strict": 72.0, "Soft": 76.96}, "OCR": {"Strict": 62.0, "Soft": 85.18}, "Document QA": {"Strict": 67.0, "Soft": 81.55}, "Structured extraction": {"Strict": 35.0, "Soft": 89.37}}, "columns": {"Provider": "OpenAI"}}, {"id": "GPT-5.6-Luna", "tags": [], "timestamp": "17/07/2026,14:45:31", "results": {"History and culture": {"Strict": 57.0, "Soft": 65.18}, "Contemporary life": {"Strict": 62.0, "Soft": 68.41}, "Geography and environment": {"Strict": 61.0, "Soft": 73.07}, "OCR": {"Strict": 59.0, "Soft": 82.93}, "Document QA": {"Strict": 60.0, "Soft": 74.43}, "Structured extraction": {"Strict": 31.0, "Soft": 87.82}}, "columns": {"Provider": "OpenAI"}}, {"id": "Kimi-K3", "tags": [], "timestamp": "17/07/2026,21:35:39", "results": {"History and culture": {"Strict": 55.0, "Soft": 61.79}, "Contemporary life": {"Strict": 57.0, "Soft": 64.87}, "Geography and environment": {"Strict": 60.0, "Soft": 71.84}, "OCR": {"Strict": 80.0, "Soft": 92.57}, "Document QA": {"Strict": 79.0, "Soft": 89.07}, "Structured extraction": {"Strict": 54.0, "Soft": 93.78}}, "columns": {"Provider": "Moonshot.AI"}}, {"id": "Gemini-3.6-Flash", "tags": [], "timestamp": "22/07/2026,10:57:04", "results": {"History and culture": {"Strict": 79.0, "Soft": 86.33}, "Contemporary life": {"Strict": 82.0, "Soft": 87.11}, "Geography and environment": {"Strict": 85.0, "Soft": 91.75}, "ASR": {"Strict": 65.0, "Soft": 87.16}, "Speech QA": {"Strict": 71.0, "Soft": 80.38}, "Sound and music QA": {"Strict": 39.0, "Soft": 46.0}, "OCR": {"Strict": 85.0, "Soft": 94.94}, "Document QA": {"Strict": 86.0, "Soft": 92.51}, "Structured extraction": {"Strict": 71.0, "Soft": 97.05}}, "columns": {"Provider": "Google"}}, {"id": "Gemini-3.5-Flash-Lite", "tags": [], "timestamp": "22/07/2026,11:04:23", "results": {"History and culture": {"Strict": 49.0, "Soft": 59.77}, "Contemporary life": {"Strict": 44.0, "Soft": 56.0}, "Geography and environment": {"Strict": 49.0, "Soft": 61.98}, "ASR": {"Strict": 43.0, "Soft": 82.79}, "Speech QA": {"Strict": 20.0, "Soft": 35.13}, "Sound and music QA": {"Strict": 20.0, "Soft": 27.22}, "OCR": {"Strict": 75.0, "Soft": 91.05}, "Document QA": {"Strict": 43.0, "Soft": 62.32}, "Structured extraction": {"Strict": 48.0, "Soft": 93.65}}, "columns": {"Provider": "Google"}}, {"id": "Inkling", "tags": [], "timestamp": "23/07/2026,14:41:33", "results": {"History and culture": {"Strict": 53.0, "Soft": 57.9}, "Contemporary life": {"Strict": 46.0, "Soft": 57.11}, "Geography and environment": {"Strict": 48.0, "Soft": 59.13}, "ASR": {"Strict": 37.0, "Soft": 63.83}, "Speech QA": {"Strict": 38.0, "Soft": 47.79}, "Sound and music QA": {"Strict": 16.0, "Soft": 23.88}, "OCR": {"Strict": 56.0, "Soft": 80.5}, "Document QA": {"Strict": 53.0, "Soft": 71.4}, "Structured extraction": {"Strict": 36.0, "Soft": 82.91}}, "columns": {"Provider": "TM"}}, {"id": "Gemini-3.7-Flash", "tags": [], "timestamp": "17/08/2026,08:06:51", "results": {"History and culture": {"Strict": 85.0, "Soft": 89.09}, "Contemporary life": {"Strict": 83.0, "Soft": 89.09}, "Geography and environment": {"Strict": 89.0, "Soft": 93.53}, "ASR": {"Strict": 78.0, "Soft": 90.59}, "Speech QA": {"Strict": 84.0, "Soft": 89.27}, "Sound and music QA": {"Strict": 45.0, "Soft": 49.83}, "OCR": {"Strict": 90.0, "Soft": 96.52}, "Document QA": {"Strict": 87.0, "Soft": 94.91}, "Structured extraction": {"Strict": 70.0, "Soft": 97.44}}, "columns": {"Provider": "Google"}}, {"id": "Muse-Glimmer-30b", "tags": [], "timestamp": "17/08/2026,10:21:12", "results": {"History and culture": {"Strict": 46.0, "Soft": 55.77}, "Contemporary life": {"Strict": 30.0, "Soft": 44.24}, "Geography and environment": {"Strict": 37.0, "Soft": 50.06}, "OCR": {"Strict": 65.0, "Soft": 87.0}, "Document QA": {"Strict": 73.0, "Soft": 84.5}, "Structured extraction": {"Strict": 48.0, "Soft": 92.06}}, "columns": {"Provider": "Meta"}}, {"id": "Muse-Spark-1.2", "tags": [], "timestamp": "17/08/2026,10:46:09", "results": {"History and culture": {"Strict": 70.0, "Soft": 75.61}, "Contemporary life": {"Strict": 70.0, "Soft": 77.8}, "Geography and environment": {"Strict": 69.0, "Soft": 80.49}, "ASR": {"Strict": 61.0, "Soft": 84.18}, "Speech QA": {"Strict": 64.0, "Soft": 74.62}, "Sound and music QA": {"Strict": 53.0, "Soft": 58.2}, "OCR": {"Strict": 84.0, "Soft": 92.94}, "Document QA": {"Strict": 81.0, "Soft": 90.92}, "Structured extraction": {"Strict": 66.0, "Soft": 93.59}}, "columns": {"Provider": "Meta"}}, {"id": "Inkling-Small", "tags": [], "timestamp": "17/08/2026,11:45:20", "results": {"History and culture": {"Strict": 40.0, "Soft": 49.2}, "Contemporary life": {"Strict": 25.0, "Soft": 34.77}, "Geography and environment": {"Strict": 39.0, "Soft": 49.74}, "ASR": {"Strict": 25.0, "Soft": 59.88}, "Speech QA": {"Strict": 30.0, "Soft": 39.16}, "Sound and music QA": {"Strict": 16.0, "Soft": 21.85}, "OCR": {"Strict": 48.0, "Soft": 74.39}, "Document QA": {"Strict": 60.0, "Soft": 76.9}, "Structured extraction": {"Strict": 23.0, "Soft": 79.1}}, "columns": {"Provider": "TM"}}, {"id": "Grok-4.6", "tags": [], "timestamp": "17/08/2026,12:38:20", "results": {"History and culture": {"Strict": 63.0, "Soft": 69.69}, "Contemporary life": {"Strict": 48.0, "Soft": 60.82}, "Geography and environment": {"Strict": 64.0, "Soft": 74.73}, "OCR": {"Strict": 81.0, "Soft": 94.03}, "Document QA": {"Strict": 78.0, "Soft": 85.1}, "Structured extraction": {"Strict": 60.0, "Soft": 95.88}}, "columns": {"Provider": "xAI"}}], "variants": [{"tab": "Full benchmark", "lead": "All 9 categories, 20 models with complete results", "taskGroupIds": ["Images", "Audio", "Documents"], "excludeTags": []}, {"tab": "Vision only", "lead": "6 vision categories (Images + Documents), 65 models", "taskGroupIds": ["Images", "Documents"], "excludeTags": ["main"]}]}