diff --git "a/nist_comparison_3d.html" "b/nist_comparison_3d.html" new file mode 100644--- /dev/null +++ "b/nist_comparison_3d.html" @@ -0,0 +1,490 @@ + + + +CAD Benchmark — NIST (claude-opus-4-7, gemini-3.1-pro-preview, gpt-5.4) + + + + + + + + +
+

CAD Benchmark

+
+
+
Text input
+
+
Reproduce the geometry as accurately as possible from the drawing. See the drawing for all dimensions and geometric relationships. Note: the drawing shows feature dimensions in millimeters (e.g. Ø35, Ø25, Ø20 mm). The overall bounding box of the finished part is approximately 800 × 450 × 150 mm.
+
+
+
Image input
+
+drawing +
+
+
Ground truth
+
+
+
+
+ +
+
+
+ +
+
+
Time: 8min
+
Cost: $6.8
+
Tokens: 339k
+
+
+
claude-opus-4-7
+
+
+
+ +
+
+
Time: 31min
+
Cost: $2.9
+
Tokens: 501k
+
+
+
gemini-3.1-pro-preview
+
+
+
+ +
+
+
Time: 22min
+
Cost: $1.8
+
Tokens: 421k
+
+
+
gpt-5.4
+
+
+
+
+

CAD Benchmark

+
+
+
Text input
+
+
Reproduce the geometry as accurately as possible from the drawing. See the drawing for all dimensions and geometric relationships. Note: This is an inch-dimensioned drawing. Your output must be in millimeters. The overall bounding box of the finished part is approximately 303 × 485 × 163 mm.
+
+
+
Image input
+
+drawing +
+
+
Ground truth
+
+
+
+
+ +
+
+
+ +
+
+
Time: 6min
+
Cost: $3.5
+
Tokens: 121k
+
+
+
claude-opus-4-7
+
+
+
+ +
+
+
Time: 32min
+
Cost: $3.4
+
Tokens: 343k
+
+
+
gemini-3.1-pro-preview
+
+
+
+ +
+
+
Time: 19min
+
Cost: $1.6
+
Tokens: 345k
+
+
+
gpt-5.4
+
+
+
+
+

CAD Benchmark

+
+
+
Text input
+
+
Reproduce the geometry as accurately as possible from the drawing. See the drawing for all dimensions and geometric relationships. Note: This is an inch-dimensioned drawing. Your output must be in millimeters. The overall bounding box of the finished part is approximately 527 × 559 × 483 mm.
+
+
+
Image input
+
+drawing +
+
+
Ground truth
+
+
+
+
+ +
+
+
+ +
+
+
Time: 2min
+
Cost: $1.5
+
Tokens: 68k
+
+
+
claude-opus-4-7
+
+
+
+ +
+
+
Time: 28min
+
Cost: $3.0
+
Tokens: 424k
+
+
+
gemini-3.1-pro-preview
+
+
+
+ +
+
+
Time: 4min
+
Cost: $0.47
+
Tokens: 117k
+
+
+
gpt-5.4
+
+
+
+ + \ No newline at end of file