Generation Speed Benchmark
Speed
| Model | Workload | Batch | Max new | Samples | Status | Prompt bytes | Prefill ms/chunk | Prefill bytes/s | Prefill units/s | Decode ms/chunk | Decode bytes/s | Decode units/s | Total bytes/s | Speedup vs B=1 | Peak VRAM MB |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bolmo | code/1024B | 1 | 64 | 384 | ok | 472.1 | 68.8 | 6866.1 | 6880.6 | 4836.2 | 110.4 | 110.4 | 100.4 | 1.0 | 2938.8 |
| bolmo | code/1024B | 1 | 128 | 384 | ok | 472.1 | 44.8 | 10547.7 | 10570.1 | 5402.8 | 110.6 | 110.6 | 101.0 | 1.0 | 2938.8 |
| bolmo | code/1024B | 2 | 64 | 384 | ok | 472.1 | 144.4 | 6539.5 | 6553.4 | 8605.5 | 146.7 | 146.7 | 129.5 | 1.29 | 3027.4 |
| bolmo | code/1024B | 2 | 128 | 384 | ok | 472.1 | 47.3 | 19952.9 | 19995.1 | 9472.7 | 146.7 | 146.7 | 130.8 | 1.29 | 3027.4 |
| bolmo | code/1024B | 4 | 64 | 384 | ok | 472.1 | 188.2 | 10030.6 | 10051.9 | 13762.9 | 213.0 | 213.0 | 178.3 | 1.78 | 3229.9 |
| bolmo | code/1024B | 4 | 128 | 384 | ok | 472.1 | 51.5 | 36681.4 | 36759.1 | 14978.1 | 212.1 | 212.1 | 178.8 | 1.77 | 3229.9 |
| bolmo | code/1024B | 8 | 64 | 384 | ok | 472.1 | 387.7 | 9741.6 | 9762.2 | 20813.7 | 319.9 | 319.9 | 238.0 | 2.37 | 3616.3 |
| bolmo | code/1024B | 8 | 128 | 384 | ok | 472.1 | 76.8 | 49167.3 | 49271.5 | 22267.3 | 320.7 | 320.7 | 240.1 | 2.38 | 3616.3 |
| bolmo | code/1024B | 16 | 64 | 384 | ok | 472.1 | 452.1 | 16706.1 | 16741.5 | 28969.9 | 501.5 | 501.5 | 298.8 | 2.98 | 4407.8 |
| bolmo | code/1024B | 16 | 128 | 384 | ok | 472.1 | 138.9 | 54365.5 | 54480.7 | 31091.6 | 499.2 | 499.2 | 296.2 | 2.93 | 4407.8 |
| bolmo | code/256B | 1 | 64 | 384 | ok | 196.7 | 43.8 | 4490.7 | 4513.6 | 2372.5 | 109.0 | 109.0 | 99.3 | 1.0 | 2854.9 |
| bolmo | code/256B | 1 | 128 | 384 | ok | 196.7 | 44.0 | 4465.3 | 4488.0 | 2964.5 | 108.8 | 108.8 | 99.4 | 1.0 | 2858.6 |
| bolmo | code/256B | 2 | 64 | 384 | ok | 196.7 | 72.1 | 5455.3 | 5483.0 | 3815.8 | 145.9 | 145.9 | 130.2 | 1.31 | 2896.6 |
| bolmo | code/256B | 2 | 128 | 384 | ok | 196.7 | 46.9 | 8392.3 | 8435.0 | 4676.2 | 146.2 | 146.2 | 131.3 | 1.32 | 2896.6 |
| bolmo | code/256B | 4 | 64 | 384 | ok | 196.7 | 70.6 | 11143.7 | 11200.4 | 5184.4 | 220.7 | 220.7 | 191.2 | 1.93 | 2965.1 |
| bolmo | code/256B | 4 | 128 | 384 | ok | 196.7 | 52.1 | 15110.2 | 15187.0 | 6306.6 | 221.1 | 221.1 | 192.1 | 1.93 | 2966.3 |
| bolmo | code/256B | 8 | 64 | 384 | ok | 196.7 | 118.2 | 13311.2 | 13378.9 | 6677.2 | 355.1 | 355.1 | 287.1 | 2.89 | 3064.1 |
| bolmo | code/256B | 8 | 128 | 384 | ok | 196.7 | 53.5 | 29410.2 | 29559.7 | 8085.7 | 354.6 | 354.6 | 288.0 | 2.9 | 3102.8 |
| bolmo | code/256B | 16 | 64 | 384 | ok | 196.7 | 181.9 | 17296.9 | 17384.9 | 8502.9 | 578.0 | 578.0 | 410.5 | 4.13 | 3308.9 |
| bolmo | code/256B | 16 | 128 | 384 | ok | 196.7 | 68.3 | 46070.8 | 46305.0 | 10256.4 | 578.3 | 578.3 | 411.2 | 4.14 | 3374.6 |
| hnet | code/1024B | 1 | 64 | 384 | ok | 472.1 | 62.2 | 7595.3 | 7611.4 | 774.4 | 82.6 | 82.6 | 76.1 | 1.0 | 3518.6 |
| hnet | code/1024B | 1 | 128 | 384 | ok | 472.1 | 62.1 | 7601.1 | 7617.2 | 1574.6 | 81.3 | 81.3 | 78.0 | 1.0 | 3533.7 |
| hnet | code/1024B | 2 | 64 | 384 | ok | 472.1 | 64.7 | 14589.9 | 14620.8 | 1080.6 | 118.5 | 118.5 | 110.8 | 1.46 | 3801.8 |
| hnet | code/1024B | 2 | 128 | 384 | ok | 472.1 | 64.0 | 14754.4 | 14785.6 | 2190.2 | 116.9 | 116.9 | 113.0 | 1.45 | 3849.3 |
| hnet | code/1024B | 4 | 64 | 384 | ok | 472.1 | 68.2 | 27690.1 | 27748.7 | 1481.8 | 172.8 | 172.8 | 163.2 | 2.14 | 4378.3 |
| hnet | code/1024B | 4 | 128 | 384 | ok | 472.1 | 67.2 | 28092.9 | 28152.4 | 2985.5 | 171.5 | 171.5 | 166.6 | 2.14 | 4437.3 |
| hnet | code/1024B | 8 | 64 | 384 | ok | 472.1 | 82.0 | 46036.4 | 46133.9 | 1892.0 | 270.6 | 270.6 | 254.8 | 3.35 | 5463.0 |
| hnet | code/1024B | 8 | 128 | 384 | ok | 472.1 | 78.1 | 48370.5 | 48473.0 | 3832.7 | 267.2 | 267.2 | 259.5 | 3.33 | 5579.6 |
| hnet | code/1024B | 16 | 64 | 384 | ok | 472.1 | 103.5 | 72962.8 | 73117.3 | 2234.0 | 458.4 | 458.4 | 429.2 | 5.64 | 7649.6 |
| hnet | code/1024B | 16 | 128 | 384 | ok | 472.1 | 101.8 | 74199.7 | 74356.9 | 4465.0 | 458.7 | 458.7 | 443.7 | 5.69 | 7885.6 |
| hnet | code/256B | 1 | 64 | 384 | ok | 196.7 | 65.4 | 3005.8 | 3021.1 | 805.0 | 79.5 | 79.5 | 73.3 | 1.0 | 3316.6 |
| hnet | code/256B | 1 | 128 | 384 | ok | 196.7 | 61.7 | 3187.4 | 3203.6 | 1610.2 | 79.5 | 79.5 | 76.4 | 1.0 | 3339.2 |
| hnet | code/256B | 2 | 64 | 384 | ok | 196.7 | 64.0 | 6143.2 | 6174.4 | 1128.4 | 113.4 | 113.4 | 107.0 | 1.46 | 3409.3 |
| hnet | code/256B | 2 | 128 | 384 | ok | 196.7 | 62.9 | 6258.4 | 6290.2 | 2248.0 | 113.9 | 113.9 | 110.5 | 1.45 | 3441.1 |
| hnet | code/256B | 4 | 64 | 384 | ok | 196.7 | 65.9 | 11943.4 | 12004.1 | 1509.6 | 169.6 | 169.6 | 161.6 | 2.2 | 3606.9 |
| hnet | code/256B | 4 | 128 | 384 | ok | 196.7 | 66.2 | 11882.8 | 11943.2 | 3046.2 | 168.1 | 168.1 | 163.9 | 2.14 | 3665.3 |
| hnet | code/256B | 8 | 64 | 384 | ok | 196.7 | 72.9 | 21582.5 | 21692.3 | 1907.4 | 268.4 | 268.4 | 256.3 | 3.5 | 3976.9 |
| hnet | code/256B | 8 | 128 | 384 | ok | 196.7 | 72.7 | 21647.2 | 21757.3 | 3853.7 | 265.7 | 265.7 | 259.4 | 3.39 | 4092.1 |
| hnet | code/256B | 16 | 64 | 384 | ok | 196.7 | 86.7 | 36276.7 | 36461.2 | 2250.8 | 454.9 | 454.9 | 431.9 | 5.89 | 4716.5 |
| hnet | code/256B | 16 | 128 | 384 | ok | 196.7 | 86.7 | 36304.2 | 36488.8 | 4488.7 | 456.3 | 456.3 | 443.9 | 5.81 | 4952.5 |
| hnet_serial | code/1024B | 1 | 64 | 384 | ok | 472.1 | 60.2 | 7838.1 | 7854.7 | 769.5 | 83.2 | 83.2 | 76.6 | 1.0 | 3518.0 |
| hnet_serial | code/1024B | 1 | 128 | 384 | ok | 472.1 | 59.7 | 7905.9 | 7922.7 | 1556.9 | 82.2 | 82.2 | 78.9 | 1.0 | 3532.7 |
| hnet_serial | code/1024B | 2 | 64 | 384 | ok | 472.1 | 117.4 | 8042.9 | 8059.9 | 1546.0 | 82.8 | 82.8 | 76.5 | 1.0 | 3698.3 |
| hnet_serial | code/1024B | 2 | 128 | 384 | ok | 472.1 | 116.7 | 8090.6 | 8107.8 | 3108.5 | 82.4 | 82.4 | 79.1 | 1.0 | 3735.8 |
| hnet_serial | code/1024B | 4 | 64 | 384 | ok | 472.1 | 227.7 | 8291.5 | 8309.1 | 3056.5 | 83.8 | 83.8 | 77.6 | 1.01 | 3698.3 |
| hnet_serial | code/1024B | 4 | 128 | 384 | ok | 472.1 | 229.7 | 8219.7 | 8237.1 | 6225.2 | 82.2 | 82.2 | 79.1 | 1.0 | 3735.8 |
| hnet_serial | code/1024B | 8 | 64 | 384 | ok | 472.1 | 451.3 | 8367.9 | 8385.6 | 6135.1 | 83.5 | 83.5 | 77.4 | 1.01 | 3698.3 |
| hnet_serial | code/1024B | 8 | 128 | 384 | ok | 472.1 | 451.5 | 8364.2 | 8381.9 | 12405.5 | 82.5 | 82.5 | 79.5 | 1.01 | 3735.8 |
| hnet_serial | code/1024B | 16 | 64 | 384 | ok | 472.1 | 895.0 | 8438.8 | 8456.7 | 12254.5 | 83.6 | 83.6 | 77.6 | 1.01 | 3699.0 |
| hnet_serial | code/1024B | 16 | 128 | 384 | ok | 472.1 | 895.7 | 8432.5 | 8450.3 | 24806.4 | 82.6 | 82.6 | 79.5 | 1.01 | 3735.2 |
| hnet_serial | code/256B | 1 | 64 | 384 | ok | 196.7 | 65.8 | 2989.4 | 3004.6 | 798.1 | 80.2 | 80.2 | 73.7 | 1.0 | 3314.1 |
| hnet_serial | code/256B | 1 | 128 | 384 | ok | 196.7 | 59.2 | 3324.8 | 3341.7 | 1594.4 | 80.3 | 80.3 | 77.2 | 1.0 | 3338.8 |
| hnet_serial | code/256B | 2 | 64 | 384 | ok | 196.7 | 115.7 | 3400.6 | 3417.9 | 1587.2 | 80.6 | 80.6 | 74.8 | 1.01 | 3397.0 |
| hnet_serial | code/256B | 2 | 128 | 384 | ok | 196.7 | 116.1 | 3386.8 | 3404.0 | 3191.0 | 80.2 | 80.2 | 77.2 | 1.0 | 3426.6 |
| hnet_serial | code/256B | 4 | 64 | 384 | ok | 196.7 | 226.5 | 3474.1 | 3491.7 | 3186.1 | 80.3 | 80.3 | 74.8 | 1.01 | 3397.0 |
| hnet_serial | code/256B | 4 | 128 | 384 | ok | 196.7 | 226.3 | 3476.8 | 3494.5 | 6365.5 | 80.4 | 80.4 | 77.5 | 1.0 | 3426.6 |
| hnet_serial | code/256B | 8 | 64 | 384 | ok | 196.7 | 448.5 | 3508.4 | 3526.2 | 6371.8 | 80.4 | 80.4 | 74.9 | 1.02 | 3397.0 |
| hnet_serial | code/256B | 8 | 128 | 384 | ok | 196.7 | 448.8 | 3506.1 | 3524.0 | 12761.3 | 80.2 | 80.2 | 77.4 | 1.0 | 3426.6 |
| hnet_serial | code/256B | 16 | 64 | 384 | ok | 196.7 | 887.2 | 3546.8 | 3564.8 | 12710.5 | 80.6 | 80.6 | 75.1 | 1.02 | 3397.0 |
| hnet_serial | code/256B | 16 | 128 | 384 | ok | 196.7 | 888.2 | 3543.1 | 3561.2 | 25442.9 | 80.5 | 80.5 | 77.7 | 1.01 | 3426.6 |
| pythia | code/1024B | 1 | 64 | 384 | ok | 472.1 | 15.4 | 30579.0 | 8838.6 | 587.1 | 389.3 | 109.0 | 379.3 | 1.0 | 2014.9 |
| pythia | code/1024B | 1 | 128 | 384 | ok | 472.1 | 15.2 | 31063.0 | 8978.5 | 1172.3 | 387.7 | 109.2 | 382.7 | 1.0 | 2014.9 |
| pythia | code/1024B | 2 | 64 | 384 | ok | 472.1 | 18.8 | 50344.4 | 14551.6 | 593.8 | 697.0 | 208.7 | 675.6 | 1.78 | 2088.9 |
| pythia | code/1024B | 2 | 128 | 384 | ok | 472.1 | 18.4 | 51420.9 | 14862.7 | 1199.3 | 676.4 | 202.5 | 666.2 | 1.74 | 2088.9 |
| pythia | code/1024B | 4 | 64 | 384 | ok | 472.1 | 19.6 | 96476.0 | 27885.5 | 604.9 | 1293.3 | 409.4 | 1252.7 | 3.3 | 2244.1 |
| pythia | code/1024B | 4 | 128 | 384 | ok | 472.1 | 19.8 | 95362.1 | 27563.5 | 1230.2 | 1243.9 | 397.4 | 1224.2 | 3.2 | 2244.1 |
| pythia | code/1024B | 8 | 64 | 384 | ok | 472.1 | 31.4 | 120427.6 | 34808.5 | 653.0 | 2353.4 | 770.8 | 2245.5 | 5.92 | 2544.6 |
| pythia | code/1024B | 8 | 128 | 384 | ok | 472.1 | 31.9 | 118556.1 | 34267.5 | 1295.0 | 2306.7 | 763.7 | 2251.2 | 5.88 | 2544.6 |
| pythia | code/1024B | 16 | 64 | 384 | ok | 472.1 | 63.2 | 119473.2 | 34532.6 | 638.2 | 4639.1 | 1570.3 | 4220.7 | 11.13 | 3153.5 |
| pythia | code/1024B | 16 | 128 | 384 | ok | 472.1 | 62.4 | 121004.4 | 34975.2 | 1273.0 | 4533.8 | 1543.7 | 4321.7 | 11.29 | 3153.5 |
| pythia | code/256B | 1 | 64 | 384 | ok | 196.7 | 12.9 | 15205.4 | 4704.4 | 585.0 | 365.5 | 108.1 | 357.6 | 1.0 | 1959.7 |
| pythia | code/256B | 1 | 128 | 384 | ok | 196.7 | 13.0 | 15153.2 | 4688.3 | 1157.3 | 359.9 | 108.7 | 355.9 | 1.0 | 1966.7 |
| pythia | code/256B | 2 | 64 | 384 | ok | 196.7 | 18.9 | 20759.3 | 6422.8 | 597.1 | 698.3 | 204.9 | 676.8 | 1.89 | 1996.4 |
| pythia | code/256B | 2 | 128 | 384 | ok | 196.7 | 18.6 | 21139.2 | 6540.3 | 1186.2 | 684.0 | 202.2 | 673.5 | 1.89 | 2001.3 |
| pythia | code/256B | 4 | 64 | 384 | ok | 196.7 | 18.3 | 42908.1 | 13275.5 | 600.9 | 1291.7 | 401.0 | 1253.4 | 3.51 | 2023.1 |
| pythia | code/256B | 4 | 128 | 384 | ok | 196.7 | 17.3 | 45482.3 | 14072.0 | 1210.3 | 1232.1 | 387.5 | 1214.7 | 3.41 | 2057.8 |
| pythia | code/256B | 8 | 64 | 384 | ok | 196.7 | 19.6 | 80095.9 | 24781.2 | 621.1 | 2436.6 | 775.5 | 2361.8 | 6.61 | 2109.2 |
| pythia | code/256B | 8 | 128 | 384 | ok | 196.7 | 19.4 | 81135.3 | 25102.8 | 1236.0 | 2357.2 | 759.3 | 2320.8 | 6.52 | 2167.8 |
| pythia | code/256B | 16 | 64 | 384 | ok | 196.7 | 24.8 | 126664.5 | 39189.3 | 623.6 | 4867.8 | 1503.2 | 4681.0 | 13.09 | 2279.2 |
| pythia | code/256B | 16 | 128 | 384 | ok | 196.7 | 25.3 | 124556.5 | 38537.1 | 1266.8 | 4524.9 | 1406.5 | 4436.3 | 12.47 | 2398.3 |
Warmup
| Model | Workload | Batch | Max new | Samples | Status | Prompt bytes | Prefill ms/chunk | Decode ms/chunk | Total ms/chunk | Peak VRAM MB |
|---|---|---|---|---|---|---|---|---|---|---|
| bolmo | code/1024B | 1 | 64 | 1 | ok | 265.0 | 43.8 | 3052.0 | 3341.1 | 2866.2 |
| bolmo | code/1024B | 1 | 128 | 1 | ok | 265.0 | 42.4 | 3651.7 | 3987.2 | 2857.4 |
| bolmo | code/1024B | 2 | 64 | 2 | ok | 613.5 | 1894.2 | 13495.6 | 17069.0 | 3026.0 |
| bolmo | code/1024B | 2 | 128 | 2 | ok | 613.5 | 38.1 | 14306.3 | 16128.1 | 3026.0 |
| bolmo | code/1024B | 4 | 64 | 4 | ok | 531.5 | 56.9 | 18632.9 | 22371.1 | 3217.5 |
| bolmo | code/1024B | 4 | 128 | 4 | ok | 531.5 | 50.8 | 19793.1 | 23814.4 | 3217.5 |
| bolmo | code/1024B | 8 | 64 | 8 | ok | 425.2 | 1960.2 | 24538.3 | 35218.8 | 3572.6 |
| bolmo | code/1024B | 8 | 128 | 8 | ok | 425.2 | 83.9 | 26349.0 | 35921.7 | 3572.6 |
| bolmo | code/1024B | 16 | 64 | 16 | ok | 423.9 | 2017.6 | 25250.7 | 43045.2 | 4319.6 |
| bolmo | code/1024B | 16 | 128 | 16 | ok | 423.9 | 146.6 | 27043.3 | 44540.5 | 4319.6 |
| bolmo | code/256B | 1 | 64 | 1 | ok | 212.0 | 80.8 | 2573.0 | 5345.6 | 2848.9 |
| bolmo | code/256B | 1 | 128 | 1 | ok | 212.0 | 38.8 | 3186.3 | 3472.0 | 2848.9 |
| bolmo | code/256B | 2 | 64 | 2 | ok | 196.0 | 2709.1 | 3626.6 | 6705.1 | 2876.3 |
| bolmo | code/256B | 2 | 128 | 2 | ok | 196.0 | 40.4 | 4453.0 | 4954.2 | 2876.3 |
| bolmo | code/256B | 4 | 64 | 4 | ok | 187.8 | 55.1 | 4618.7 | 5335.7 | 2924.8 |
| bolmo | code/256B | 4 | 128 | 4 | ok | 187.8 | 117.3 | 5647.9 | 6577.9 | 2924.8 |
| bolmo | code/256B | 8 | 64 | 8 | ok | 190.2 | 3023.3 | 6481.8 | 10735.6 | 3026.6 |
| bolmo | code/256B | 8 | 128 | 8 | ok | 190.2 | 49.6 | 6992.3 | 8624.1 | 3026.6 |
| bolmo | code/256B | 16 | 64 | 16 | ok | 179.8 | 3332.2 | 9968.8 | 16233.8 | 3239.8 |
| bolmo | code/256B | 16 | 128 | 16 | ok | 179.8 | 60.9 | 9696.9 | 13438.3 | 3252.2 |
| hnet | code/1024B | 1 | 64 | 1 | ok | 265.0 | 63.9 | 884.9 | 953.0 | 3319.5 |
| hnet | code/1024B | 1 | 128 | 1 | ok | 265.0 | 61.1 | 1819.8 | 1883.8 | 3343.8 |
| hnet | code/1024B | 2 | 64 | 2 | ok | 613.5 | 64.6 | 1288.2 | 1362.7 | 3773.8 |
| hnet | code/1024B | 2 | 128 | 2 | ok | 613.5 | 64.0 | 2531.4 | 2608.4 | 3826.2 |
| hnet | code/1024B | 4 | 64 | 4 | ok | 531.5 | 69.6 | 1531.7 | 1614.6 | 4310.4 |
| hnet | code/1024B | 4 | 128 | 4 | ok | 531.5 | 66.9 | 3024.8 | 3114.0 | 4370.2 |
| hnet | code/1024B | 8 | 64 | 8 | ok | 425.2 | 74.7 | 1765.8 | 1865.1 | 5350.4 |
| hnet | code/1024B | 8 | 128 | 8 | ok | 425.2 | 77.9 | 3662.8 | 3776.5 | 5470.1 |
| hnet | code/1024B | 16 | 64 | 16 | ok | 423.9 | 94.4 | 2196.2 | 2341.5 | 7480.3 |
| hnet | code/1024B | 16 | 128 | 16 | ok | 423.9 | 103.2 | 4417.9 | 4566.6 | 7716.3 |
| hnet | code/256B | 1 | 64 | 1 | ok | 212.0 | 20030.1 | 1047.3 | 21080.4 | 3556.9 |
| hnet | code/256B | 1 | 128 | 1 | ok | 212.0 | 61.1 | 1683.1 | 1747.1 | 3318.4 |
| hnet | code/256B | 2 | 64 | 2 | ok | 196.0 | 207.4 | 1191.1 | 1402.4 | 3384.2 |
| hnet | code/256B | 2 | 128 | 2 | ok | 196.0 | 63.0 | 2332.0 | 2399.4 | 3414.9 |
| hnet | code/256B | 4 | 64 | 4 | ok | 187.8 | 68.3 | 1503.7 | 1578.2 | 3559.3 |
| hnet | code/256B | 4 | 128 | 4 | ok | 187.8 | 66.7 | 3104.5 | 3182.8 | 3616.0 |
| hnet | code/256B | 8 | 64 | 8 | ok | 190.2 | 76.7 | 1872.6 | 1960.9 | 3903.5 |
| hnet | code/256B | 8 | 128 | 8 | ok | 190.2 | 73.2 | 3763.2 | 3853.8 | 4020.3 |
| hnet | code/256B | 16 | 64 | 16 | ok | 179.8 | 85.6 | 2464.0 | 2571.8 | 4640.5 |
| hnet | code/256B | 16 | 128 | 16 | ok | 179.8 | 85.2 | 4508.5 | 4627.2 | 4876.5 |
| hnet_serial | code/1024B | 1 | 64 | 1 | ok | 265.0 | 752.1 | 890.2 | 1646.6 | 3317.7 |
| hnet_serial | code/1024B | 1 | 128 | 1 | ok | 265.0 | 59.6 | 1813.3 | 1877.3 | 3343.6 |
| hnet_serial | code/1024B | 2 | 64 | 2 | ok | 613.5 | 117.1 | 1838.9 | 1967.3 | 3598.2 |
| hnet_serial | code/1024B | 2 | 128 | 2 | ok | 613.5 | 115.7 | 3463.3 | 3591.9 | 3635.6 |
| hnet_serial | code/1024B | 4 | 64 | 4 | ok | 531.5 | 225.7 | 3236.4 | 3479.5 | 3608.5 |
| hnet_serial | code/1024B | 4 | 128 | 4 | ok | 531.5 | 229.7 | 6386.6 | 6634.0 | 3635.6 |
| hnet_serial | code/1024B | 8 | 64 | 8 | ok | 425.2 | 452.2 | 6272.0 | 6749.1 | 3608.5 |
| hnet_serial | code/1024B | 8 | 128 | 8 | ok | 425.2 | 450.3 | 12604.3 | 13080.9 | 3635.6 |
| hnet_serial | code/1024B | 16 | 64 | 16 | ok | 423.9 | 894.5 | 12325.2 | 13261.0 | 3608.5 |
| hnet_serial | code/1024B | 16 | 128 | 16 | ok | 423.9 | 900.3 | 25133.6 | 26075.5 | 3635.6 |
| hnet_serial | code/256B | 1 | 64 | 1 | ok | 212.0 | 30196.8 | 847.5 | 31047.6 | 3554.1 |
| hnet_serial | code/256B | 1 | 128 | 1 | ok | 212.0 | 58.6 | 1740.8 | 1803.3 | 3317.0 |
| hnet_serial | code/256B | 2 | 64 | 2 | ok | 196.0 | 116.2 | 1691.3 | 1814.3 | 3370.4 |
| hnet_serial | code/256B | 2 | 128 | 2 | ok | 196.0 | 116.4 | 3369.0 | 3499.2 | 3398.4 |
| hnet_serial | code/256B | 4 | 64 | 4 | ok | 187.8 | 226.9 | 3276.3 | 3513.7 | 3370.4 |
| hnet_serial | code/256B | 4 | 128 | 4 | ok | 187.8 | 225.6 | 6515.2 | 6751.9 | 3398.4 |
| hnet_serial | code/256B | 8 | 64 | 8 | ok | 190.2 | 447.1 | 6504.2 | 6969.9 | 3376.2 |
| hnet_serial | code/256B | 8 | 128 | 8 | ok | 190.2 | 449.7 | 12838.5 | 13306.9 | 3404.0 |
| hnet_serial | code/256B | 16 | 64 | 16 | ok | 179.8 | 884.6 | 13329.9 | 14247.4 | 3382.4 |
| hnet_serial | code/256B | 16 | 128 | 16 | ok | 179.8 | 887.1 | 26288.8 | 27209.4 | 3413.2 |
| pythia | code/1024B | 1 | 64 | 1 | ok | 265.0 | 16.3 | 589.7 | 606.0 | 1959.9 |
| pythia | code/1024B | 1 | 128 | 1 | ok | 265.0 | 13.7 | 1162.3 | 1176.1 | 1966.8 |
| pythia | code/1024B | 2 | 64 | 2 | ok | 613.5 | 34.4 | 610.7 | 645.2 | 2065.1 |
| pythia | code/1024B | 2 | 128 | 2 | ok | 613.5 | 19.9 | 1176.4 | 1196.4 | 2065.1 |
| pythia | code/1024B | 4 | 64 | 4 | ok | 531.5 | 20.4 | 602.2 | 622.7 | 2192.2 |
| pythia | code/1024B | 4 | 128 | 4 | ok | 531.5 | 20.1 | 1202.6 | 1222.8 | 2192.2 |
| pythia | code/1024B | 8 | 64 | 8 | ok | 425.2 | 32.5 | 615.5 | 648.0 | 2438.1 |
| pythia | code/1024B | 8 | 128 | 8 | ok | 425.2 | 37.0 | 1230.5 | 1267.5 | 2438.1 |
| pythia | code/1024B | 16 | 64 | 16 | ok | 423.9 | 59.5 | 639.4 | 698.9 | 2944.8 |
| pythia | code/1024B | 16 | 128 | 16 | ok | 423.9 | 65.0 | 1277.3 | 1342.3 | 2944.8 |
| pythia | code/256B | 1 | 64 | 1 | ok | 212.0 | 18.7 | 597.2 | 616.0 | 1957.2 |
| pythia | code/256B | 1 | 128 | 1 | ok | 212.0 | 12.8 | 1176.5 | 1189.4 | 1965.2 |
| pythia | code/256B | 2 | 64 | 2 | ok | 196.0 | 21.6 | 599.2 | 620.8 | 1996.1 |
| pythia | code/256B | 2 | 128 | 2 | ok | 196.0 | 18.8 | 1179.8 | 1198.6 | 1998.0 |
| pythia | code/256B | 4 | 64 | 4 | ok | 187.8 | 16.3 | 597.9 | 614.2 | 2014.2 |
| pythia | code/256B | 4 | 128 | 4 | ok | 187.8 | 16.2 | 1197.6 | 1213.8 | 2053.4 |
| pythia | code/256B | 8 | 64 | 8 | ok | 190.2 | 19.3 | 615.4 | 634.7 | 2088.5 |
| pythia | code/256B | 8 | 128 | 8 | ok | 190.2 | 19.0 | 1233.3 | 1252.3 | 2156.6 |
| pythia | code/256B | 16 | 64 | 16 | ok | 179.8 | 22.3 | 624.2 | 646.5 | 2239.5 |
| pythia | code/256B | 16 | 128 | 16 | ok | 179.8 | 23.7 | 1272.3 | 1296.0 | 2382.3 |
Compression
| Model | Workload | Raw prompt bytes | Model input units | Stage0 / patches | Stage1 | Bytes per stage0 | Stage0 per stage1 |
|---|---|---|---|---|---|---|---|
| bolmo | code/1024B | 181272.0 | 181656.0 | 40221.0 | None | 4.51 | None |
| bolmo | code/1024B | 181272.0 | 181656.0 | 40221.0 | None | 4.51 | None |
| bolmo | code/1024B | 181272.0 | 181656.0 | 40206.0 | None | 4.51 | None |
| bolmo | code/1024B | 181272.0 | 181656.0 | 40206.0 | None | 4.51 | None |
| bolmo | code/1024B | 181272.0 | 181656.0 | 40194.0 | None | 4.51 | None |
| bolmo | code/1024B | 181272.0 | 181656.0 | 40194.0 | None | 4.51 | None |
| bolmo | code/1024B | 181272.0 | 181656.0 | 40212.0 | None | 4.51 | None |
| bolmo | code/1024B | 181272.0 | 181656.0 | 40212.0 | None | 4.51 | None |
| bolmo | code/1024B | 181272.0 | 181656.0 | 40209.0 | None | 4.51 | None |
| bolmo | code/1024B | 181272.0 | 181656.0 | 40209.0 | None | 4.51 | None |
| bolmo | code/256B | 75525.0 | 75909.0 | 17772.0 | None | 4.25 | None |
| bolmo | code/256B | 75525.0 | 75909.0 | 17772.0 | None | 4.25 | None |
| bolmo | code/256B | 75525.0 | 75909.0 | 17769.0 | None | 4.25 | None |
| bolmo | code/256B | 75525.0 | 75909.0 | 17769.0 | None | 4.25 | None |
| bolmo | code/256B | 75525.0 | 75909.0 | 17769.0 | None | 4.25 | None |
| bolmo | code/256B | 75525.0 | 75909.0 | 17769.0 | None | 4.25 | None |
| bolmo | code/256B | 75525.0 | 75909.0 | 17775.0 | None | 4.25 | None |
| bolmo | code/256B | 75525.0 | 75909.0 | 17775.0 | None | 4.25 | None |
| bolmo | code/256B | 75525.0 | 75909.0 | 17763.0 | None | 4.25 | None |
| bolmo | code/256B | 75525.0 | 75909.0 | 17763.0 | None | 4.25 | None |
| hnet | code/1024B | 181272.0 | 181656.0 | 63561.0 | 30735.0 | 2.85 | 2.07 |
| hnet | code/1024B | 181272.0 | 181656.0 | 63561.0 | 30735.0 | 2.85 | 2.07 |
| hnet | code/1024B | 181272.0 | 181656.0 | 63564.0 | 30726.0 | 2.85 | 2.07 |
| hnet | code/1024B | 181272.0 | 181656.0 | 63564.0 | 30726.0 | 2.85 | 2.07 |
| hnet | code/1024B | 181272.0 | 181656.0 | 63576.0 | 30720.0 | 2.85 | 2.07 |
| hnet | code/1024B | 181272.0 | 181656.0 | 63576.0 | 30720.0 | 2.85 | 2.07 |
| hnet | code/1024B | 181272.0 | 181656.0 | 63573.0 | 30723.0 | 2.85 | 2.07 |
| hnet | code/1024B | 181272.0 | 181656.0 | 63573.0 | 30723.0 | 2.85 | 2.07 |
| hnet | code/1024B | 181272.0 | 181656.0 | 63570.0 | 30729.0 | 2.85 | 2.07 |
| hnet | code/1024B | 181272.0 | 181656.0 | 63570.0 | 30729.0 | 2.85 | 2.07 |
| hnet | code/256B | 75525.0 | 75909.0 | 28476.0 | 14922.0 | 2.65 | 1.91 |
| hnet | code/256B | 75525.0 | 75909.0 | 28476.0 | 14922.0 | 2.65 | 1.91 |
| hnet | code/256B | 75525.0 | 75909.0 | 28479.0 | 14916.0 | 2.65 | 1.91 |
| hnet | code/256B | 75525.0 | 75909.0 | 28479.0 | 14916.0 | 2.65 | 1.91 |
| hnet | code/256B | 75525.0 | 75909.0 | 28473.0 | 14919.0 | 2.65 | 1.91 |
| hnet | code/256B | 75525.0 | 75909.0 | 28473.0 | 14919.0 | 2.65 | 1.91 |
| hnet | code/256B | 75525.0 | 75909.0 | 28473.0 | 14922.0 | 2.65 | 1.91 |
| hnet | code/256B | 75525.0 | 75909.0 | 28473.0 | 14922.0 | 2.65 | 1.91 |
| hnet | code/256B | 75525.0 | 75909.0 | 28473.0 | 14928.0 | 2.65 | 1.91 |
| hnet | code/256B | 75525.0 | 75909.0 | 28473.0 | 14928.0 | 2.65 | 1.91 |
| hnet_serial | code/1024B | 181272.0 | 181656.0 | 63561.0 | 30726.0 | 2.85 | 2.07 |
| hnet_serial | code/1024B | 181272.0 | 181656.0 | 63561.0 | 30726.0 | 2.85 | 2.07 |
| hnet_serial | code/1024B | 181272.0 | 181656.0 | 63561.0 | 30726.0 | 2.85 | 2.07 |
| hnet_serial | code/1024B | 181272.0 | 181656.0 | 63561.0 | 30726.0 | 2.85 | 2.07 |
| hnet_serial | code/1024B | 181272.0 | 181656.0 | 63561.0 | 30726.0 | 2.85 | 2.07 |
| hnet_serial | code/1024B | 181272.0 | 181656.0 | 63561.0 | 30726.0 | 2.85 | 2.07 |
| hnet_serial | code/1024B | 181272.0 | 181656.0 | 63561.0 | 30726.0 | 2.85 | 2.07 |
| hnet_serial | code/1024B | 181272.0 | 181656.0 | 63561.0 | 30726.0 | 2.85 | 2.07 |
| hnet_serial | code/1024B | 181272.0 | 181656.0 | 63561.0 | 30726.0 | 2.85 | 2.07 |
| hnet_serial | code/1024B | 181272.0 | 181656.0 | 63561.0 | 30726.0 | 2.85 | 2.07 |
| hnet_serial | code/256B | 75525.0 | 75909.0 | 28476.0 | 14922.0 | 2.65 | 1.91 |
| hnet_serial | code/256B | 75525.0 | 75909.0 | 28476.0 | 14922.0 | 2.65 | 1.91 |
| hnet_serial | code/256B | 75525.0 | 75909.0 | 28476.0 | 14922.0 | 2.65 | 1.91 |
| hnet_serial | code/256B | 75525.0 | 75909.0 | 28476.0 | 14922.0 | 2.65 | 1.91 |
| hnet_serial | code/256B | 75525.0 | 75909.0 | 28476.0 | 14922.0 | 2.65 | 1.91 |
| hnet_serial | code/256B | 75525.0 | 75909.0 | 28476.0 | 14922.0 | 2.65 | 1.91 |
| hnet_serial | code/256B | 75525.0 | 75909.0 | 28476.0 | 14922.0 | 2.65 | 1.91 |
| hnet_serial | code/256B | 75525.0 | 75909.0 | 28476.0 | 14922.0 | 2.65 | 1.91 |
| hnet_serial | code/256B | 75525.0 | 75909.0 | 28476.0 | 14922.0 | 2.65 | 1.91 |
| hnet_serial | code/256B | 75525.0 | 75909.0 | 28476.0 | 14922.0 | 2.65 | 1.91 |
| pythia | code/1024B | 181272.0 | 52395.0 | None | None | None | None |
| pythia | code/1024B | 181272.0 | 52395.0 | None | None | None | None |
| pythia | code/1024B | 181272.0 | 52395.0 | None | None | None | None |
| pythia | code/1024B | 181272.0 | 52395.0 | None | None | None | None |
| pythia | code/1024B | 181272.0 | 52395.0 | None | None | None | None |
| pythia | code/1024B | 181272.0 | 52395.0 | None | None | None | None |
| pythia | code/1024B | 181272.0 | 52395.0 | None | None | None | None |
| pythia | code/1024B | 181272.0 | 52395.0 | None | None | None | None |
| pythia | code/1024B | 181272.0 | 52395.0 | None | None | None | None |
| pythia | code/1024B | 181272.0 | 52395.0 | None | None | None | None |
| pythia | code/256B | 75525.0 | 23367.0 | None | None | None | None |
| pythia | code/256B | 75525.0 | 23367.0 | None | None | None | None |
| pythia | code/256B | 75525.0 | 23367.0 | None | None | None | None |
| pythia | code/256B | 75525.0 | 23367.0 | None | None | None | None |
| pythia | code/256B | 75525.0 | 23367.0 | None | None | None | None |
| pythia | code/256B | 75525.0 | 23367.0 | None | None | None | None |
| pythia | code/256B | 75525.0 | 23367.0 | None | None | None | None |
| pythia | code/256B | 75525.0 | 23367.0 | None | None | None | None |
| pythia | code/256B | 75525.0 | 23367.0 | None | None | None | None |
| pythia | code/256B | 75525.0 | 23367.0 | None | None | None | None |