KucLab commited on
Commit
13f1e59
·
verified ·
1 Parent(s): dda133f

Add report benchmark.md

Browse files
Files changed (1) hide show
  1. reports/benchmark.md +41 -0
reports/benchmark.md ADDED
@@ -0,0 +1,41 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Barist.1 — Benchmark
2
+
3
+ Generated by `scripts/benchmark.py`.
4
+
5
+ ## Hardware
6
+
7
+ - Device: `AMD Radeon Graphics` (cuda)
8
+ - torch: `2.13.0+rocm7.1`
9
+ - Python: `3.12.12`
10
+ - OS: `Linux-7.1.7-100.fc43.x86_64-x86_64-with-glibc2.43`
11
+
12
+ ## Model
13
+
14
+ | Property | Value |
15
+ | --- | --- |
16
+ | Parameters | 903,552 |
17
+ | Checkpoint size | 3.45 MB |
18
+ | Vocabulary | 2048 |
19
+ | Context | 4096 tokens |
20
+ | Input limit | 4000 characters |
21
+ | Layers / d_model / heads / d_ff | 3 / 128 / 4 / 384 |
22
+
23
+ ## Inference
24
+
25
+ | Metric | Value |
26
+ | --- | --- |
27
+ | Single text latency | 1.04 ms |
28
+ | Single text throughput | 958.1 texts/s |
29
+ | Batch of 8 latency | 1.23 ms |
30
+ | Batched throughput | 6490.0 texts/s |
31
+ | Peak VRAM (inference) | 84.7 MB |
32
+
33
+ ## Accuracy
34
+
35
+ | Split | Rows | Accuracy | F1 | Precision | Recall |
36
+ | --- | --- | --- | --- | --- | --- |
37
+ | val | 6585 | 0.8535 | 0.8175 | 0.8498 | 0.7875 |
38
+ | test | 6496 | 0.8570 | 0.8225 | 0.8519 | 0.7950 |
39
+ | probe (hand-written) | 97 | 0.9897 | 0.9882 | 0.9767 | 1.0000 |
40
+
41
+ The probe set is hand-written and absent from all training data.