ashish-negi-sima-ai commited on
Commit
7de92ee
·
verified ·
1 Parent(s): 333aed7

Fix accuracy benchmark commands

Browse files
Files changed (1) hide show
  1. README.md +15 -9
README.md CHANGED
@@ -106,23 +106,29 @@ scp -r LFM2-2.6B-a16w4_return_logits sima@<modalix-ip>:/media/nvme/llima/models/
106
 
107
  ### Modalix Backend Accuracy
108
 
109
- Run Modalix accuracy benchmarking from a host machine using this precompiled `--return_logits` artifact:
110
 
111
  ```bash
112
- llima-benchmark accuracy LFM2-2.6B-a16w4_return_logits -b modalix -o <output_dir> --max_num_tokens <max_num_tokens> \
113
- --board_ip <board_ip> --board_port <board_port> \
114
- --board_model LFM2-2.6B-a16w4_return_logits --board_start_server \
115
- --board_venv_path <venv_on_board>
 
 
 
116
  ```
117
 
118
- The `--board_model` path must already exist on the Modalix device and contain both `devkit/` and `elf_files/`.
119
 
120
- ### HF Backend Accuracy
121
 
122
- Run the Hugging Face backend reference benchmark from the host machine:
123
 
124
  ```bash
125
- llima-benchmark accuracy LiquidAI/LFM2-2.6B -b hf -o <output_dir>
 
 
 
126
  ```
127
 
128
  See [MOLE accuracy benchmarking](https://developer.sima.ai/software/genai-llima/mole) for the full accuracy benchmarking workflow.
 
106
 
107
  ### Modalix Backend Accuracy
108
 
109
+ Run the benchmark from a host machine. The positional `model_id` must be the source Hugging Face model because MOLE uses its tokenizer and configuration; `--board_model` selects this compiled artifact on Modalix.
110
 
111
  ```bash
112
+ MODALIX_IP="<modalix-ip>"
113
+ llima-benchmark accuracy LiquidAI/LFM2-2.6B \
114
+ --backend modalix \
115
+ --output ./results/modalix \
116
+ --board_ip "$MODALIX_IP" \
117
+ --board_model LFM2-2.6B-a16w4_return_logits \
118
+ --task hellaswag piqa triviaqa wikitext winogrande
119
  ```
120
 
121
+ The compiled `--board_model` must already exist on Modalix and contain both `devkit/` and `elf_files/`. The CLI starts the benchmark server by default. Add `--board_venv_path /path/to/venv` when `llima` is installed in a non-default virtual environment, or `--no-board_start_server` when connecting to a server that is already running.
122
 
123
+ ### Hugging Face Backend Accuracy
124
 
125
+ Run the same task suite against the source model on the host:
126
 
127
  ```bash
128
+ llima-benchmark accuracy LiquidAI/LFM2-2.6B \
129
+ --backend hf \
130
+ --output ./results/hf \
131
+ --task hellaswag piqa triviaqa wikitext winogrande
132
  ```
133
 
134
  See [MOLE accuracy benchmarking](https://developer.sima.ai/software/genai-llima/mole) for the full accuracy benchmarking workflow.