Spaces:
Running
Running
| <p> | |
| A one-image smoke test of the four open-weight vision-language models that are | |
| candidates to replace Gemini in the analysis layer, run locally on the L40S | |
| (vLLM 0.31.0). Each model gets the same image, the same one-line prompt and | |
| the same strict JSON schema, at temperature 0. It checks that the model loads, | |
| reads an image and returns schema-valid JSON through the repository's own | |
| <code>vllm</code> client. It is not a quality comparison: that is the pilot on | |
| 15 papers (<a href="https://github.com/llm-art/art-historical-papers-extraction/blob/main/docs/model-pilot.md"><code>docs/model-pilot.md</code></a>). | |
| </p> | |
| <h2>Input</h2> | |
| <figure> | |
| <img src="https://huggingface.co/datasets/gspin/phd-report-media/resolve/main/figures/68590ebc5313/f001.jpg" alt="Hendrick ter Brugghen, Crucifixion with the Virgin and Saint John" loading="lazy"> | |
| <figcaption>Hendrick ter Brugghen, <em>Crucifixion with the Virgin and Saint John</em>, | |
| ca. 1625, New York, The Metropolitan Museum of Art (public domain). Figure f001 of | |
| paper <code>68590ebc5313</code> (JHNA), as extracted: 329 × 376 px.</figcaption> | |
| </figure> | |
| <p>Prompt: <code>Describe this image as JSON.</code> Schema: <code>medium</code> (string), | |
| <code>subject</code> (string), <code>people_count</code> (integer or null), | |
| <code>notes</code> (list of strings). No system prompt, no caption, no article text.</p> | |
| <h2>Results</h2> | |
| <div class="table-wrap"><table> | |
| <thead><tr><th>Model</th><th>Checkpoint</th><th>Weights</th><th>Thinking</th> | |
| <th>Valid JSON</th><th class="num">Prompt tokens</th><th class="num">Output tokens</th> | |
| <th class="num">Seconds</th></tr></thead> | |
| <tbody> | |
| <tr><td>Qwen3.6-35B-A3B</td><td><a href="https://huggingface.co/Qwen/Qwen3.6-35B-A3B-FP8/tree/95a723d"><code>Qwen/Qwen3.6-35B-A3B-FP8</code></a> <code>95a723d</code></td><td>official FP8</td><td>off</td><td>yes</td><td class="num">230</td><td class="num">157</td><td class="num">31.53</td></tr> | |
| <tr><td>Qwen3.8-27B</td><td><a href="https://huggingface.co/Qwen/Qwen3.8-27B-FP8/tree/017b9c7"><code>Qwen/Qwen3.8-27B-FP8</code></a> <code>017b9c7</code></td><td>official FP8</td><td>off</td><td>yes</td><td class="num">230</td><td class="num">177</td><td class="num">12.48</td></tr> | |
| <tr><td>Gemma 4 26B-A4B</td><td><a href="https://huggingface.co/RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic/tree/ed35d7a"><code>RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic</code></a> <code>ed35d7a</code></td><td>third-party FP8</td><td>off</td><td>yes</td><td class="num">1106</td><td class="num">100</td><td class="num">16.95</td></tr> | |
| <tr><td>Mistral Small 3.2 24B</td><td><a href="https://huggingface.co/RedHatAI/Mistral-Small-3.2-24B-Instruct-2506-FP8/tree/9bc507c"><code>RedHatAI/Mistral-Small-3.2-24B-Instruct-2506-FP8</code></a> <code>9bc507c</code></td><td>third-party FP8</td><td>n/a (not a reasoning model)</td><td>yes</td><td class="num">192</td><td class="num">166</td><td class="num">9.82</td></tr> | |
| </tbody> | |
| </table></div> | |
| <div class="callout warning"> | |
| <p>Seconds are wall-clock and include waiting behind the pilot's own requests on the | |
| same server (one request at a time), so they are not a speed measurement. Prompt | |
| tokens are mostly the image: models size images differently (Qwen by a pixel range, | |
| Gemma by a token budget, Mistral by a 1,540 px cap), so equal images do not cost | |
| equal tokens.</p> | |
| </div> | |
| <h2>Answers</h2> | |
| <h3>Qwen3.6-35B-A3B</h3> | |
| <pre><code class="language-json">{ | |
| "medium": "oil painting", | |
| "subject": "Crucifixion of Jesus Christ", | |
| "people_count": 3, | |
| "notes": [ | |
| "Jesus is depicted on the cross, with a white loincloth and dark hair.", | |
| "To the left, a woman in a grey hooded robe kneels in grief, head bowed.", | |
| "To the right, a man in red and dark robes looks up at Jesus with an expression of sorrow or devotion.", | |
| "The background is dark and atmospheric, with subtle vertical lines suggesting rain or light rays.", | |
| "The composition is vertical, emphasizing the height of the cross and the emotional weight of the scene." | |
| ] | |
| }</code></pre> | |
| <h3>Qwen3.8-27B</h3> | |
| <pre><code class="language-json">{ | |
| "medium": "painting", | |
| "subject": "The Crucifixion of Jesus Christ", | |
| "people_count": 3, | |
| "notes": [ | |
| "The central figure is Jesus on the cross, depicted with a gaunt, emaciated body and a bowed head.", | |
| "To the left, a figure in a white hooded robe is shown in profile, looking down with hands clasped in prayer or grief.", | |
| "To the right, a man in a red robe looks up at Jesus, holding his foot.", | |
| "The overall color palette is dark and somber, dominated by browns, blacks, and muted earth tones, with the red robe providing a stark contrast.", | |
| "The style is dramatic and expressive, characteristic of the Baroque period." | |
| ] | |
| }</code></pre> | |
| <h3>Gemma 4 26B-A4B</h3> | |
| <pre><code class="language-json">{ | |
| "medium": "oil on canvas", | |
| "subject": "The Crucifixion of Jesus Christ", | |
| "people_count": 3, | |
| "notes": [ | |
| "The central figure is Jesus Christ on the cross.", | |
| "To the left is a woman, likely the Virgin Mary, draped in white and blue robes.", | |
| "To the right is a man, likely Saint John the Apostle, wearing a red cloak.", | |
| "The background is dark and dramatic, typical of Baroque art style." | |
| ] | |
| }</code></pre> | |
| <h3>Mistral Small 3.2 24B</h3> | |
| <pre><code class="language-json">{ | |
| "medium": "Painting", | |
| "subject": "Religious", | |
| "people_count": 3, | |
| "notes": [ | |
| "This image depicts a religious scene, specifically the crucifixion of Jesus Christ.", | |
| "The central figure is Jesus, nailed to a cross with his arms outstretched.", | |
| "Two other figures are present at the base of the cross, one on each side. The figure on the left appears to be a woman, possibly the Virgin Mary, and the figure on the right is a man, possibly John the Apostle.", | |
| "The painting uses a dark, dramatic color palette, emphasizing the somber and tragic nature of the scene.", | |
| "The background is dark and cloudy, adding to the overall mood of the painting." | |
| ] | |
| }</code></pre> | |
| <h2>Notes</h2> | |
| <ul> | |
| <li>Gemma and Mistral run from RedHat's FP8 builds because the official bf16 weights do | |
| not fit in 48 GB, so their output includes quantisation error.</li> | |
| <li>Mistral's tokenizer rejects the <code>enable_thinking</code> switch the Qwen and Gemma | |
| models need (HTTP 400), so it is called without it; it has no thinking mode.</li> | |
| <li>Each server is started with <code>make serve-vllm VLLM_PRESET=<preset></code>, | |
| pinned to the sha above.</li> | |
| </ul> | |