Card: image timings on an idle NPU, 31k row
Browse files- MANIFEST.sha256 +1 -1
- README.md +7 -5
MANIFEST.sha256
CHANGED
|
@@ -4,7 +4,7 @@
|
|
| 4 |
c71d239df91726fc519c6eb72d318ec65820627232b2f796219e87dcf35d0ab4 LICENSE
|
| 5 |
4527ccda731d2c1149d8529d46ed6ce4035f19041951ba87b579d9dfe867bb33 model.q4nx
|
| 6 |
b926ac0e9a72a24e29e3f518f3241064416bc137c185c3349c3eae2370cfe7a0 NOTICE
|
| 7 |
-
|
| 8 |
662e8a26d8016b2d479f1ca66bf64944603316c2b5e074ab7ac3304b8ae045e9 readout.safetensors
|
| 9 |
49e2b6e395f959f077f1e992b338919c0d4a9732fc6e613995e06557f843500c tokenizer_config.json
|
| 10 |
5f9e4d4901a92b997e463c1f46055088b6cca5ca61a6522d1b9f64c4bb81cb42 tokenizer.json
|
|
|
|
| 4 |
c71d239df91726fc519c6eb72d318ec65820627232b2f796219e87dcf35d0ab4 LICENSE
|
| 5 |
4527ccda731d2c1149d8529d46ed6ce4035f19041951ba87b579d9dfe867bb33 model.q4nx
|
| 6 |
b926ac0e9a72a24e29e3f518f3241064416bc137c185c3349c3eae2370cfe7a0 NOTICE
|
| 7 |
+
b1d7211158ed046b23f571e34b9ad9239b90291a313a8714a817960c594a6074 README.md
|
| 8 |
662e8a26d8016b2d479f1ca66bf64944603316c2b5e074ab7ac3304b8ae045e9 readout.safetensors
|
| 9 |
49e2b6e395f959f077f1e992b338919c0d4a9732fc6e613995e06557f843500c tokenizer_config.json
|
| 10 |
5f9e4d4901a92b997e463c1f46055088b6cca5ca61a6522d1b9f64c4bb81cb42 tokenizer.json
|
README.md
CHANGED
|
@@ -60,16 +60,18 @@ suite; the Decision Index public vision suite has not been run for this package.
|
|
| 60 |
| 1 question, 1,024 tokens | 584 ms |
|
| 61 |
| 1 question, 4,096 tokens | 2,470 ms |
|
| 62 |
| 1 question, 15,854 tokens | 11.4 s |
|
|
|
|
| 63 |
| 1 question, 32,768 tokens | 28.4 s |
|
| 64 |
| 4 questions, 330 tokens each | 796 ms |
|
| 65 |
| 16 questions, 330 tokens each | 2,889 ms |
|
| 66 |
-
| 1 question + a 1024 x 768 image (876 tokens, HTTP) |
|
| 67 |
-
| 1 question +
|
| 68 |
-
|
|
|
|
|
| 69 |
|
| 70 |
-
From 1,536 tokens attention runs on the NPU's attention kernel (below that the CPU is faster). Prompts over 32,768
|
| 71 |
tokens are answered with `max_length_exceeded` (nothing is truncated). Text rows were measured at the engine (HTTP
|
| 72 |
-
adds about a millisecond); the image rows over HTTP.
|
| 73 |
|
| 74 |
## License
|
| 75 |
|
|
|
|
| 60 |
| 1 question, 1,024 tokens | 584 ms |
|
| 61 |
| 1 question, 4,096 tokens | 2,470 ms |
|
| 62 |
| 1 question, 15,854 tokens | 11.4 s |
|
| 63 |
+
| 1 question, 31,491 tokens | 27.1 s |
|
| 64 |
| 1 question, 32,768 tokens | 28.4 s |
|
| 65 |
| 4 questions, 330 tokens each | 796 ms |
|
| 66 |
| 16 questions, 330 tokens each | 2,889 ms |
|
| 67 |
+
| 1 question + a 1024 x 768 image (876 tokens, HTTP) | 1.3 s |
|
| 68 |
+
| 1 question + two images (HTTP) | 2.4 s |
|
| 69 |
+
| 1 question + a 2000 x 1500 image (downscaled to 1.6 MP, HTTP) | 3.0 s |
|
| 70 |
+
| 8 questions + one image (one image encoding, HTTP) | 5.0 s |
|
| 71 |
|
| 72 |
+
From 1,536 tokens attention runs on the NPU's attention kernel (below that the CPU is faster; at 31,491 tokens it is 3.3x faster than attention on the CPU). Prompts over 32,768
|
| 73 |
tokens are answered with `max_length_exceeded` (nothing is truncated). Text rows were measured at the engine (HTTP
|
| 74 |
+
adds about a millisecond); the image rows over HTTP, one question, median of 3, on an otherwise idle NPU.
|
| 75 |
|
| 76 |
## License
|
| 77 |
|