Image-Text-to-Text
Transformers
Safetensors
inkling2
conversational
Mixture of Experts
Eval Results
8-bit precision
modelopt
Instructions to use thinkingmachines/Inkling-2-Small with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use thinkingmachines/Inkling-2-Small with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="thinkingmachines/Inkling-2-Small") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("thinkingmachines/Inkling-2-Small") model = AutoModelForMultimodalLM.from_pretrained("thinkingmachines/Inkling-2-Small", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Inference
- HuggingChat
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use thinkingmachines/Inkling-2-Small with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "thinkingmachines/Inkling-2-Small" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "thinkingmachines/Inkling-2-Small", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/thinkingmachines/Inkling-2-Small
- SGLang
How to use thinkingmachines/Inkling-2-Small with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "thinkingmachines/Inkling-2-Small" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "thinkingmachines/Inkling-2-Small", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "thinkingmachines/Inkling-2-Small" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "thinkingmachines/Inkling-2-Small", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use thinkingmachines/Inkling-2-Small with Docker Model Runner:
docker model run hf.co/thinkingmachines/Inkling-2-Small
[marstin-dev] drop sage1.1 for mistral4l on validation set; update tab headline metric
#7
by marstin-thinky - opened
README.md
CHANGED
|
@@ -146,7 +146,7 @@ The training data curation process includes cleaning, processing, and modifying
|
|
| 146 |
<tr><td style="background-color:rgba(128,128,128,0.1);font-weight:600;border-right-color:rgba(128,128,128,0.45)">Finance Work</td><td style="background-color:rgba(128,128,128,0.1);border-right-color:rgba(128,128,128,0.45)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1);border-right-color:rgba(128,128,128,0.45)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1);border-right-color:rgba(128,128,128,0.45)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td></tr>
|
| 147 |
<tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45)">Finance Agent v2.1</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)"><span title="Best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.75);color:#282828;font-weight:600">67.5%</span></td><td style="text-align:center">60.6%</td><td style="text-align:center">63.3%</td><td style="text-align:center">58.8%</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)">62.9%</td><td style="text-align:center">60.7%</td><td style="text-align:center">65.0%</td><td style="text-align:center">64.3%</td><td style="text-align:center">61.7%</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)">60.0%</td><td style="text-align:center">56.8%</td><td style="text-align:center"><span title="Second-best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.28);color:inherit">65.8%</span></td><td style="text-align:center">65.4%</td></tr>
|
| 148 |
<tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45)">Excel Modeling Benchmark v1.1</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)">69.5%</td><td style="text-align:center">66.3%</td><td style="text-align:center">68.9%</td><td style="text-align:center">67.8%</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)">69.2%</td><td style="text-align:center">62.0%</td><td style="text-align:center">69.1%</td><td style="text-align:center">67.4%</td><td style="text-align:center">68.7%</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)"><span title="Second-best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.28);color:inherit">71.3%</span></td><td style="text-align:center">70.0%</td><td style="text-align:center"><span title="Best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.75);color:#282828;font-weight:600">72.9%</span></td><td style="text-align:center">65.5%</td></tr>
|
| 149 |
-
<tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45)">Tax Agent Bench</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)">
|
| 150 |
<tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45);vertical-align:middle">Apex-Accounting v1.1<br><span style="font-size:11px;opacity:0.7">pass@1</span></td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)">14.5%</td><td style="text-align:center;vertical-align:middle">12.3%</td><td style="text-align:center;vertical-align:middle">–</td><td style="text-align:center;vertical-align:middle">13.5%</td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)">12.5%</td><td style="text-align:center;vertical-align:middle">13.8%</td><td style="text-align:center;vertical-align:middle">13.9%</td><td style="text-align:center;vertical-align:middle">–</td><td style="text-align:center;vertical-align:middle">12.0%</td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)">9.5%</td><td style="text-align:center;vertical-align:middle">10.8%</td><td style="text-align:center;vertical-align:middle"><span title="Best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.75);color:#282828;font-weight:600">15.5%</span></td><td style="text-align:center;vertical-align:middle"><span title="Second-best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.28);color:inherit">15.3%</span></td></tr>
|
| 151 |
<tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45)">DAYJOB (Finance)</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)"><span title="Best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.75);color:#282828;font-weight:600">14.3%</span></td><td style="text-align:center">7.3%</td><td style="text-align:center">–</td><td style="text-align:center">–</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)"><span title="Second-best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.28);color:inherit">12.5%</span></td><td style="text-align:center">–</td><td style="text-align:center">11.0%</td><td style="text-align:center">–</td><td style="text-align:center">4.8%</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)">5.5%</td><td style="text-align:center">6.8%</td><td style="text-align:center">4.5%</td><td style="text-align:center">–</td></tr>
|
| 152 |
<tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45)">DeepFinance</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)"><span title="Second-best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.28);color:inherit">29.8%</span></td><td style="text-align:center">–</td><td style="text-align:center">–</td><td style="text-align:center">–</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)">–</td><td style="text-align:center">–</td><td style="text-align:center">27.5%</td><td style="text-align:center">–</td><td style="text-align:center">21.5%</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)">–</td><td style="text-align:center">26.4%</td><td style="text-align:center">–</td><td style="text-align:center"><span title="Best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.75);color:#282828;font-weight:600">40.0%</span></td></tr>
|
|
@@ -179,7 +179,7 @@ The training data curation process includes cleaning, processing, and modifying
|
|
| 179 |
<tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45);vertical-align:middle">Charxiv RQ<br><span style="font-size:11px;opacity:0.7">original / with Python</span></td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)"><span title="Second-best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.28);color:inherit">85.6/89.1%</span></td><td style="text-align:center;vertical-align:middle">80.2/89.4%</td><td style="text-align:center;vertical-align:middle">66.2/88.4%</td><td style="text-align:center;vertical-align:middle">82.5/87.6%</td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)">77.0/89.6%</td><td style="text-align:center;vertical-align:middle">85.4/89.5%</td><td style="text-align:center;vertical-align:middle">–</td><td style="text-align:center;vertical-align:middle">74.2/86.7%</td><td style="text-align:center;vertical-align:middle">83.2/87.4%</td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)">84.8/91.3%</td><td style="text-align:center;vertical-align:middle">81.8/87.6%</td><td style="text-align:center;vertical-align:middle"><span title="Best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.75);color:#282828;font-weight:600">86.2/92.4%</span></td><td style="text-align:center;vertical-align:middle">84.4/86.3%</td></tr>
|
| 180 |
<tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45);vertical-align:middle">Chartography<br><span style="font-size:11px;opacity:0.7">original / with Python</span></td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)">36.7/82.1%</td><td style="text-align:center;vertical-align:middle">16.3/78.0%</td><td style="text-align:center;vertical-align:middle">7.5/73.9%</td><td style="text-align:center;vertical-align:middle">12.6%/–</td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)">17.9/83.0%</td><td style="text-align:center;vertical-align:middle">35.7/65.8%</td><td style="text-align:center;vertical-align:middle">–</td><td style="text-align:center;vertical-align:middle">10.8/69.0%</td><td style="text-align:center;vertical-align:middle">21.2/55.2%</td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)">26.6/66.6%</td><td style="text-align:center;vertical-align:middle">32.0/66.5%</td><td style="text-align:center;vertical-align:middle"><span title="Second-best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.28);color:inherit">40.9/81.0%</span></td><td style="text-align:center;vertical-align:middle"><span title="Best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.75);color:#282828;font-weight:600">46.4/86.2%</span></td></tr>
|
| 181 |
<tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45);vertical-align:middle">VIALS<br><span style="font-size:11px;opacity:0.7">original / with stirrup v0.2.1</span></td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)"><span title="Best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.75);color:#282828;font-weight:600">34.6/57.8%</span></td><td style="text-align:center;vertical-align:middle">–/53.0%</td><td style="text-align:center;vertical-align:middle">–</td><td style="text-align:center;vertical-align:middle">–/49.7%</td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)">–/53.2%</td><td style="text-align:center;vertical-align:middle">–</td><td style="text-align:center;vertical-align:middle">–</td><td style="text-align:center;vertical-align:middle">–/58.4%</td><td style="text-align:center;vertical-align:middle">–/42.0%</td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)"><span title="Second-best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.28);color:inherit">20.9%/–</span></td><td style="text-align:center;vertical-align:middle">–/44.1%</td><td style="text-align:center;vertical-align:middle">–/52.0%</td><td style="text-align:center;vertical-align:middle">–/60.5%</td></tr>
|
| 182 |
-
<tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45)">SAGE v1.1</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)">88.6%</td><td style="text-align:center"><span title="Second-best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.28);color:inherit">89.3%</span></td><td style="text-align:center">83.3%</td><td style="text-align:center">85.3%</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)">–</td><td style="text-align:center">85.8%</td><td style="text-align:center">–</td><td style="text-align:center">84.4%</td><td style="text-align:center">
|
| 183 |
<tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45)">MuirBench</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)"><span title="Second-best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.28);color:inherit">79.2%</span></td><td style="text-align:center">71.6%</td><td style="text-align:center">63.2%</td><td style="text-align:center">69.4%</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)">75.7%</td><td style="text-align:center">74.5%</td><td style="text-align:center">–</td><td style="text-align:center">72.6%</td><td style="text-align:center">67.5%</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)">77.0%</td><td style="text-align:center">73.9%</td><td style="text-align:center"><span title="Best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.75);color:#282828;font-weight:600">80.1%</span></td><td style="text-align:center">74.0%</td></tr>
|
| 184 |
<tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45);vertical-align:middle">ZeroBench (main)<br><span style="font-size:11px;opacity:0.7">original / with Python</span></td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)"><span title="Best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.75);color:#282828;font-weight:600">29.0/44.0%</span></td><td style="text-align:center;vertical-align:middle">13.0/32.0%</td><td style="text-align:center;vertical-align:middle">7.0%/–</td><td style="text-align:center;vertical-align:middle">21.0%/–</td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)"><span title="Second-best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.28);color:inherit">23.0%/–</span></td><td style="text-align:center;vertical-align:middle">14.0/50.0%</td><td style="text-align:center;vertical-align:middle">–</td><td style="text-align:center;vertical-align:middle">8.0%/–</td><td style="text-align:center;vertical-align:middle">15.0/23.0%</td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)"><span title="Second-best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.28);color:inherit">23.0/41.0%</span></td><td style="text-align:center;vertical-align:middle">16.0%/–</td><td style="text-align:center;vertical-align:middle"><span title="Second-best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.28);color:inherit">23.0%/–</span></td><td style="text-align:center;vertical-align:middle"><span title="Second-best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.28);color:inherit">23.0%/–</span></td></tr>
|
| 185 |
<tr><td style="background-color:rgba(128,128,128,0.1);font-weight:600;border-right-color:rgba(128,128,128,0.45)">STEM Reasoning</td><td style="background-color:rgba(128,128,128,0.1);border-right-color:rgba(128,128,128,0.45)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1);border-right-color:rgba(128,128,128,0.45)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1);border-right-color:rgba(128,128,128,0.45)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td></tr>
|
|
|
|
| 146 |
<tr><td style="background-color:rgba(128,128,128,0.1);font-weight:600;border-right-color:rgba(128,128,128,0.45)">Finance Work</td><td style="background-color:rgba(128,128,128,0.1);border-right-color:rgba(128,128,128,0.45)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1);border-right-color:rgba(128,128,128,0.45)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1);border-right-color:rgba(128,128,128,0.45)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td></tr>
|
| 147 |
<tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45)">Finance Agent v2.1</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)"><span title="Best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.75);color:#282828;font-weight:600">67.5%</span></td><td style="text-align:center">60.6%</td><td style="text-align:center">63.3%</td><td style="text-align:center">58.8%</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)">62.9%</td><td style="text-align:center">60.7%</td><td style="text-align:center">65.0%</td><td style="text-align:center">64.3%</td><td style="text-align:center">61.7%</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)">60.0%</td><td style="text-align:center">56.8%</td><td style="text-align:center"><span title="Second-best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.28);color:inherit">65.8%</span></td><td style="text-align:center">65.4%</td></tr>
|
| 148 |
<tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45)">Excel Modeling Benchmark v1.1</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)">69.5%</td><td style="text-align:center">66.3%</td><td style="text-align:center">68.9%</td><td style="text-align:center">67.8%</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)">69.2%</td><td style="text-align:center">62.0%</td><td style="text-align:center">69.1%</td><td style="text-align:center">67.4%</td><td style="text-align:center">68.7%</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)"><span title="Second-best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.28);color:inherit">71.3%</span></td><td style="text-align:center">70.0%</td><td style="text-align:center"><span title="Best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.75);color:#282828;font-weight:600">72.9%</span></td><td style="text-align:center">65.5%</td></tr>
|
| 149 |
+
<tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45)">Tax Agent Bench</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)">38.9%</td><td style="text-align:center">31.6%</td><td style="text-align:center">28.5%</td><td style="text-align:center">32.1%</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)"><span title="Best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.75);color:#282828;font-weight:600">44.4%</span></td><td style="text-align:center">26.3%</td><td style="text-align:center"><span title="Second-best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.28);color:inherit">39.7%</span></td><td style="text-align:center">32.4%</td><td style="text-align:center">30.7%</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)">33.4%</td><td style="text-align:center">19.9%</td><td style="text-align:center">32.1%</td><td style="text-align:center">31.9%</td></tr>
|
| 150 |
<tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45);vertical-align:middle">Apex-Accounting v1.1<br><span style="font-size:11px;opacity:0.7">pass@1</span></td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)">14.5%</td><td style="text-align:center;vertical-align:middle">12.3%</td><td style="text-align:center;vertical-align:middle">–</td><td style="text-align:center;vertical-align:middle">13.5%</td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)">12.5%</td><td style="text-align:center;vertical-align:middle">13.8%</td><td style="text-align:center;vertical-align:middle">13.9%</td><td style="text-align:center;vertical-align:middle">–</td><td style="text-align:center;vertical-align:middle">12.0%</td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)">9.5%</td><td style="text-align:center;vertical-align:middle">10.8%</td><td style="text-align:center;vertical-align:middle"><span title="Best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.75);color:#282828;font-weight:600">15.5%</span></td><td style="text-align:center;vertical-align:middle"><span title="Second-best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.28);color:inherit">15.3%</span></td></tr>
|
| 151 |
<tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45)">DAYJOB (Finance)</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)"><span title="Best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.75);color:#282828;font-weight:600">14.3%</span></td><td style="text-align:center">7.3%</td><td style="text-align:center">–</td><td style="text-align:center">–</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)"><span title="Second-best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.28);color:inherit">12.5%</span></td><td style="text-align:center">–</td><td style="text-align:center">11.0%</td><td style="text-align:center">–</td><td style="text-align:center">4.8%</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)">5.5%</td><td style="text-align:center">6.8%</td><td style="text-align:center">4.5%</td><td style="text-align:center">–</td></tr>
|
| 152 |
<tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45)">DeepFinance</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)"><span title="Second-best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.28);color:inherit">29.8%</span></td><td style="text-align:center">–</td><td style="text-align:center">–</td><td style="text-align:center">–</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)">–</td><td style="text-align:center">–</td><td style="text-align:center">27.5%</td><td style="text-align:center">–</td><td style="text-align:center">21.5%</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)">–</td><td style="text-align:center">26.4%</td><td style="text-align:center">–</td><td style="text-align:center"><span title="Best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.75);color:#282828;font-weight:600">40.0%</span></td></tr>
|
|
|
|
| 179 |
<tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45);vertical-align:middle">Charxiv RQ<br><span style="font-size:11px;opacity:0.7">original / with Python</span></td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)"><span title="Second-best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.28);color:inherit">85.6/89.1%</span></td><td style="text-align:center;vertical-align:middle">80.2/89.4%</td><td style="text-align:center;vertical-align:middle">66.2/88.4%</td><td style="text-align:center;vertical-align:middle">82.5/87.6%</td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)">77.0/89.6%</td><td style="text-align:center;vertical-align:middle">85.4/89.5%</td><td style="text-align:center;vertical-align:middle">–</td><td style="text-align:center;vertical-align:middle">74.2/86.7%</td><td style="text-align:center;vertical-align:middle">83.2/87.4%</td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)">84.8/91.3%</td><td style="text-align:center;vertical-align:middle">81.8/87.6%</td><td style="text-align:center;vertical-align:middle"><span title="Best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.75);color:#282828;font-weight:600">86.2/92.4%</span></td><td style="text-align:center;vertical-align:middle">84.4/86.3%</td></tr>
|
| 180 |
<tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45);vertical-align:middle">Chartography<br><span style="font-size:11px;opacity:0.7">original / with Python</span></td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)">36.7/82.1%</td><td style="text-align:center;vertical-align:middle">16.3/78.0%</td><td style="text-align:center;vertical-align:middle">7.5/73.9%</td><td style="text-align:center;vertical-align:middle">12.6%/–</td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)">17.9/83.0%</td><td style="text-align:center;vertical-align:middle">35.7/65.8%</td><td style="text-align:center;vertical-align:middle">–</td><td style="text-align:center;vertical-align:middle">10.8/69.0%</td><td style="text-align:center;vertical-align:middle">21.2/55.2%</td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)">26.6/66.6%</td><td style="text-align:center;vertical-align:middle">32.0/66.5%</td><td style="text-align:center;vertical-align:middle"><span title="Second-best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.28);color:inherit">40.9/81.0%</span></td><td style="text-align:center;vertical-align:middle"><span title="Best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.75);color:#282828;font-weight:600">46.4/86.2%</span></td></tr>
|
| 181 |
<tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45);vertical-align:middle">VIALS<br><span style="font-size:11px;opacity:0.7">original / with stirrup v0.2.1</span></td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)"><span title="Best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.75);color:#282828;font-weight:600">34.6/57.8%</span></td><td style="text-align:center;vertical-align:middle">–/53.0%</td><td style="text-align:center;vertical-align:middle">–</td><td style="text-align:center;vertical-align:middle">–/49.7%</td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)">–/53.2%</td><td style="text-align:center;vertical-align:middle">–</td><td style="text-align:center;vertical-align:middle">–</td><td style="text-align:center;vertical-align:middle">–/58.4%</td><td style="text-align:center;vertical-align:middle">–/42.0%</td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)"><span title="Second-best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.28);color:inherit">20.9%/–</span></td><td style="text-align:center;vertical-align:middle">–/44.1%</td><td style="text-align:center;vertical-align:middle">–/52.0%</td><td style="text-align:center;vertical-align:middle">–/60.5%</td></tr>
|
| 182 |
+
<tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45)">SAGE v1.1</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)">88.6%</td><td style="text-align:center"><span title="Second-best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.28);color:inherit">89.3%</span></td><td style="text-align:center">83.3%</td><td style="text-align:center">85.3%</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)">–</td><td style="text-align:center">85.8%</td><td style="text-align:center">–</td><td style="text-align:center">84.4%</td><td style="text-align:center">–</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)">86.8%</td><td style="text-align:center">83.4%</td><td style="text-align:center">88.2%</td><td style="text-align:center"><span title="Best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.75);color:#282828;font-weight:600">89.8%</span></td></tr>
|
| 183 |
<tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45)">MuirBench</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)"><span title="Second-best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.28);color:inherit">79.2%</span></td><td style="text-align:center">71.6%</td><td style="text-align:center">63.2%</td><td style="text-align:center">69.4%</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)">75.7%</td><td style="text-align:center">74.5%</td><td style="text-align:center">–</td><td style="text-align:center">72.6%</td><td style="text-align:center">67.5%</td><td style="text-align:center;border-right-color:rgba(128,128,128,0.45)">77.0%</td><td style="text-align:center">73.9%</td><td style="text-align:center"><span title="Best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.75);color:#282828;font-weight:600">80.1%</span></td><td style="text-align:center">74.0%</td></tr>
|
| 184 |
<tr style="white-space:nowrap"><td style="border-right-color:rgba(128,128,128,0.45);vertical-align:middle">ZeroBench (main)<br><span style="font-size:11px;opacity:0.7">original / with Python</span></td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)"><span title="Best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.75);color:#282828;font-weight:600">29.0/44.0%</span></td><td style="text-align:center;vertical-align:middle">13.0/32.0%</td><td style="text-align:center;vertical-align:middle">7.0%/–</td><td style="text-align:center;vertical-align:middle">21.0%/–</td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)"><span title="Second-best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.28);color:inherit">23.0%/–</span></td><td style="text-align:center;vertical-align:middle">14.0/50.0%</td><td style="text-align:center;vertical-align:middle">–</td><td style="text-align:center;vertical-align:middle">8.0%/–</td><td style="text-align:center;vertical-align:middle">15.0/23.0%</td><td style="text-align:center;vertical-align:middle;border-right-color:rgba(128,128,128,0.45)"><span title="Second-best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.28);color:inherit">23.0/41.0%</span></td><td style="text-align:center;vertical-align:middle">16.0%/–</td><td style="text-align:center;vertical-align:middle"><span title="Second-best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.28);color:inherit">23.0%/–</span></td><td style="text-align:center;vertical-align:middle"><span title="Second-best reported score in this row" style="display:block;margin:0 -4px;padding:2px 4px;border-radius:3px;background-color:rgba(255,206,82,0.28);color:inherit">23.0%/–</span></td></tr>
|
| 185 |
<tr><td style="background-color:rgba(128,128,128,0.1);font-weight:600;border-right-color:rgba(128,128,128,0.45)">STEM Reasoning</td><td style="background-color:rgba(128,128,128,0.1);border-right-color:rgba(128,128,128,0.45)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1);border-right-color:rgba(128,128,128,0.45)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1);border-right-color:rgba(128,128,128,0.45)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td><td style="background-color:rgba(128,128,128,0.1)"></td></tr>
|