| # Generation Speed Benchmark |
|
|
| ## Speed |
|
|
| | Model | Workload | Batch | Max new | Samples | Status | Prompt bytes | Prefill ms/chunk | Prefill bytes/s | Prefill units/s | Decode ms/chunk | Decode bytes/s | Decode units/s | Total bytes/s | Speedup vs B=1 | Peak VRAM MB | |
| |---|---|---:|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| |
| | bolmo | code/1024B | 1 | 128 | 384 | ok | 472.1 | 46.6 | 10123.8 | 10145.2 | 6104.9 | 97.9 | 97.9 | 89.6 | 1.0 | 2938.8 | |
| | bolmo | code/1024B | 16 | 128 | 384 | ok | 472.1 | 137.4 | 54989.0 | 55105.4 | 33932.4 | 457.4 | 457.4 | 276.6 | 3.09 | 4407.8 | |
| | bolmo | code/256B | 1 | 128 | 384 | ok | 196.7 | 46.4 | 4234.8 | 4256.3 | 3324.8 | 97.1 | 97.1 | 88.7 | 1.0 | 2858.6 | |
| | bolmo | code/256B | 16 | 128 | 384 | ok | 196.7 | 71.6 | 43960.8 | 44184.3 | 11234.8 | 527.9 | 527.9 | 378.0 | 4.26 | 3372.3 | |
| | hnet | code/1024B | 1 | 128 | 384 | ok | 472.1 | 67.6 | 6988.0 | 7002.8 | 1778.0 | 72.0 | 72.0 | 69.2 | 1.0 | 3533.7 | |
| | hnet | code/1024B | 16 | 128 | 384 | ok | 472.1 | 114.4 | 66038.5 | 66178.3 | 5086.3 | 402.7 | 402.7 | 390.4 | 5.64 | 7885.6 | |
| | hnet | code/256B | 1 | 128 | 384 | ok | 196.7 | 69.3 | 2836.2 | 2850.6 | 1822.3 | 70.2 | 70.2 | 67.6 | 1.0 | 3339.2 | |
| | hnet | code/256B | 16 | 128 | 384 | ok | 196.7 | 94.1 | 33448.2 | 33618.3 | 5105.2 | 401.2 | 401.2 | 390.5 | 5.78 | 4952.5 | |
| | hnet_serial | code/1024B | 1 | 128 | 384 | ok | 472.1 | 65.1 | 7249.9 | 7265.3 | 1778.6 | 72.0 | 72.0 | 69.2 | 1.0 | 3532.7 | |
| | hnet_serial | code/1024B | 16 | 128 | 384 | ok | 472.1 | 1010.5 | 7474.4 | 7490.3 | 28367.3 | 72.2 | 72.2 | 69.6 | 1.01 | 3735.2 | |
| | hnet_serial | code/256B | 1 | 128 | 384 | ok | 196.7 | 66.2 | 2971.2 | 2986.3 | 1818.8 | 70.4 | 70.4 | 67.8 | 1.0 | 3338.8 | |
| | hnet_serial | code/256B | 16 | 128 | 384 | ok | 196.7 | 999.6 | 3148.1 | 3164.1 | 29037.1 | 70.5 | 70.5 | 68.1 | 1.0 | 3426.6 | |
| | pythia | code/1024B | 1 | 128 | 384 | ok | 472.1 | 19.7 | 23958.7 | 6925.0 | 1897.7 | 242.4 | 67.5 | 239.9 | 1.0 | 2804.9 | |
| | pythia | code/1024B | 16 | 128 | 384 | ok | 472.1 | 80.1 | 94333.9 | 27266.3 | 1935.9 | 4714.0 | 1057.9 | 4526.7 | 18.87 | 4257.4 | |
| | pythia | code/256B | 1 | 128 | 384 | ok | 196.7 | 18.1 | 10884.6 | 3367.6 | 1835.5 | 222.7 | 68.2 | 220.6 | 1.0 | 2748.9 | |
| | pythia | code/256B | 16 | 128 | 384 | ok | 196.7 | 34.4 | 91458.4 | 28296.7 | 1952.6 | 3775.3 | 1009.9 | 3709.8 | 16.82 | 3389.1 | |
|
|
| ## Warmup |
|
|
| | Model | Workload | Batch | Max new | Samples | Status | Prompt bytes | Prefill ms/chunk | Decode ms/chunk | Total ms/chunk | Peak VRAM MB | |
| |---|---|---:|---:|---:|---|---:|---:|---:|---:|---:| |
| | bolmo | code/1024B | 1 | 128 | 1 | ok | 265.0 | 46.6 | 4182.5 | 4562.7 | 2866.2 | |
| | bolmo | code/1024B | 16 | 128 | 16 | ok | 423.9 | 148.4 | 29721.0 | 48056.7 | 4319.6 | |
| | bolmo | code/256B | 1 | 128 | 1 | ok | 212.0 | 76.9 | 3506.3 | 6426.9 | 2848.9 | |
| | bolmo | code/256B | 16 | 128 | 16 | ok | 179.8 | 75.7 | 10638.0 | 14660.3 | 3253.2 | |
| | hnet | code/1024B | 1 | 128 | 1 | ok | 265.0 | 70.0 | 2075.2 | 2151.1 | 3341.3 | |
| | hnet | code/1024B | 16 | 128 | 16 | ok | 423.9 | 101.4 | 4994.2 | 5108.4 | 7716.3 | |
| | hnet | code/256B | 1 | 128 | 1 | ok | 212.0 | 20130.7 | 2466.6 | 22599.6 | 3571.7 | |
| | hnet | code/256B | 16 | 128 | 16 | ok | 179.8 | 230.1 | 5088.0 | 5327.1 | 4876.5 | |
| | hnet_serial | code/1024B | 1 | 128 | 1 | ok | 265.0 | 202.9 | 2060.7 | 2268.6 | 3341.2 | |
| | hnet_serial | code/1024B | 16 | 128 | 16 | ok | 423.9 | 1008.7 | 28635.2 | 29684.3 | 3635.6 | |
| | hnet_serial | code/256B | 1 | 128 | 1 | ok | 212.0 | 8846.9 | 2008.2 | 10858.4 | 3567.3 | |
| | hnet_serial | code/256B | 16 | 128 | 16 | ok | 179.8 | 993.3 | 30005.9 | 31032.6 | 3413.2 | |
| | pythia | code/1024B | 1 | 128 | 1 | ok | 265.0 | 23.9 | 1904.3 | 1928.3 | 2749.1 | |
| | pythia | code/1024B | 16 | 128 | 16 | ok | 423.9 | 81.2 | 1959.2 | 2040.5 | 3998.7 | |
| | pythia | code/256B | 1 | 128 | 1 | ok | 212.0 | 21.1 | 1892.9 | 1914.0 | 2746.7 | |
| | pythia | code/256B | 16 | 128 | 16 | ok | 179.8 | 32.9 | 1964.5 | 1997.5 | 3368.2 | |
|
|
| ## Compression |
|
|
| | Model | Workload | Raw prompt bytes | Model input units | Stage0 / patches | Stage1 | Bytes per stage0 | Stage0 per stage1 | |
| |---|---|---:|---:|---:|---:|---:|---:| |
| | bolmo | code/1024B | 181272.0 | 181656.0 | 40221.0 | None | 4.51 | None | |
| | bolmo | code/1024B | 181272.0 | 181656.0 | 40209.0 | None | 4.51 | None | |
| | bolmo | code/256B | 75525.0 | 75909.0 | 17772.0 | None | 4.25 | None | |
| | bolmo | code/256B | 75525.0 | 75909.0 | 17763.0 | None | 4.25 | None | |
| | hnet | code/1024B | 181272.0 | 181656.0 | 63564.0 | 30714.0 | 2.85 | 2.07 | |
| | hnet | code/1024B | 181272.0 | 181656.0 | 63570.0 | 30738.0 | 2.85 | 2.07 | |
| | hnet | code/256B | 75525.0 | 75909.0 | 28470.0 | 14919.0 | 2.65 | 1.91 | |
| | hnet | code/256B | 75525.0 | 75909.0 | 28473.0 | 14928.0 | 2.65 | 1.91 | |
| | hnet_serial | code/1024B | 181272.0 | 181656.0 | 63564.0 | 30714.0 | 2.85 | 2.07 | |
| | hnet_serial | code/1024B | 181272.0 | 181656.0 | 63564.0 | 30714.0 | 2.85 | 2.07 | |
| | hnet_serial | code/256B | 75525.0 | 75909.0 | 28470.0 | 14919.0 | 2.65 | 1.91 | |
| | hnet_serial | code/256B | 75525.0 | 75909.0 | 28470.0 | 14919.0 | 2.65 | 1.91 | |
| | pythia | code/1024B | 181272.0 | 52395.0 | None | None | None | None | |
| | pythia | code/1024B | 181272.0 | 52395.0 | None | None | None | None | |
| | pythia | code/256B | 75525.0 | 23367.0 | None | None | None | None | |
| | pythia | code/256B | 75525.0 | 23367.0 | None | None | None | None | |
|
|
