FastFlowLM commited on
Commit
e144267
·
verified ·
1 Parent(s): c5f8dfa

Card: image timings on an idle NPU, 31k row

Browse files
Files changed (2) hide show
  1. MANIFEST.sha256 +1 -1
  2. README.md +7 -5
MANIFEST.sha256 CHANGED
@@ -4,7 +4,7 @@
4
  c71d239df91726fc519c6eb72d318ec65820627232b2f796219e87dcf35d0ab4 LICENSE
5
  4527ccda731d2c1149d8529d46ed6ce4035f19041951ba87b579d9dfe867bb33 model.q4nx
6
  b926ac0e9a72a24e29e3f518f3241064416bc137c185c3349c3eae2370cfe7a0 NOTICE
7
- 1673bed3ee81b7121142e744896754325c413a6b67dcc2ab03707f7738c1760b README.md
8
  662e8a26d8016b2d479f1ca66bf64944603316c2b5e074ab7ac3304b8ae045e9 readout.safetensors
9
  49e2b6e395f959f077f1e992b338919c0d4a9732fc6e613995e06557f843500c tokenizer_config.json
10
  5f9e4d4901a92b997e463c1f46055088b6cca5ca61a6522d1b9f64c4bb81cb42 tokenizer.json
 
4
  c71d239df91726fc519c6eb72d318ec65820627232b2f796219e87dcf35d0ab4 LICENSE
5
  4527ccda731d2c1149d8529d46ed6ce4035f19041951ba87b579d9dfe867bb33 model.q4nx
6
  b926ac0e9a72a24e29e3f518f3241064416bc137c185c3349c3eae2370cfe7a0 NOTICE
7
+ b1d7211158ed046b23f571e34b9ad9239b90291a313a8714a817960c594a6074 README.md
8
  662e8a26d8016b2d479f1ca66bf64944603316c2b5e074ab7ac3304b8ae045e9 readout.safetensors
9
  49e2b6e395f959f077f1e992b338919c0d4a9732fc6e613995e06557f843500c tokenizer_config.json
10
  5f9e4d4901a92b997e463c1f46055088b6cca5ca61a6522d1b9f64c4bb81cb42 tokenizer.json
README.md CHANGED
@@ -60,16 +60,18 @@ suite; the Decision Index public vision suite has not been run for this package.
60
  | 1 question, 1,024 tokens | 584 ms |
61
  | 1 question, 4,096 tokens | 2,470 ms |
62
  | 1 question, 15,854 tokens | 11.4 s |
 
63
  | 1 question, 32,768 tokens | 28.4 s |
64
  | 4 questions, 330 tokens each | 796 ms |
65
  | 16 questions, 330 tokens each | 2,889 ms |
66
- | 1 question + a 1024 x 768 image (876 tokens, HTTP) | 2.2 s |
67
- | 1 question + a 2000 x 1500 image (1,672 tokens, HTTP) | 4.8 s |
68
- | 8 questions + one such image (one image encoding, HTTP) | 8.1 s |
 
69
 
70
- From 1,536 tokens attention runs on the NPU's attention kernel (below that the CPU is faster). Prompts over 32,768
71
  tokens are answered with `max_length_exceeded` (nothing is truncated). Text rows were measured at the engine (HTTP
72
- adds about a millisecond); the image rows over HTTP.
73
 
74
  ## License
75
 
 
60
  | 1 question, 1,024 tokens | 584 ms |
61
  | 1 question, 4,096 tokens | 2,470 ms |
62
  | 1 question, 15,854 tokens | 11.4 s |
63
+ | 1 question, 31,491 tokens | 27.1 s |
64
  | 1 question, 32,768 tokens | 28.4 s |
65
  | 4 questions, 330 tokens each | 796 ms |
66
  | 16 questions, 330 tokens each | 2,889 ms |
67
+ | 1 question + a 1024 x 768 image (876 tokens, HTTP) | 1.3 s |
68
+ | 1 question + two images (HTTP) | 2.4 s |
69
+ | 1 question + a 2000 x 1500 image (downscaled to 1.6 MP, HTTP) | 3.0 s |
70
+ | 8 questions + one image (one image encoding, HTTP) | 5.0 s |
71
 
72
+ From 1,536 tokens attention runs on the NPU's attention kernel (below that the CPU is faster; at 31,491 tokens it is 3.3x faster than attention on the CPU). Prompts over 32,768
73
  tokens are answered with `max_length_exceeded` (nothing is truncated). Text rows were measured at the engine (HTTP
74
+ adds about a millisecond); the image rows over HTTP, one question, median of 3, on an otherwise idle NPU.
75
 
76
  ## License
77