Update README.md

README.md

@@ -17,6 +17,57 @@ Some reasons for using these checkpoints:
 - You can use them starting point to train your own small language model.
 - More interestingly, you can prob into the learning process of these models to understand how LLM learns to mimic human.
 
+# Evaluation results
+
+**Note** this does not represent the final performance of the model and should only be served as a reference for my training progress.
+```
+checkpoint: step-00088000
+
+|    Tasks    |Version|Filter|n-shot| Metric |Value |   |Stderr|
+|-------------|------:|------|-----:|--------|-----:|---|-----:|
+|piqa         |      1|none  |     0|acc     |0.6202|±  |0.0113|
+|             |       |none  |     0|acc_norm|0.6213|±  |0.0113|
+|boolq        |      2|none  |     0|acc     |0.5875|±  |0.0086|
+|arc_challenge|      1|none  |     0|acc     |0.1980|±  |0.0116|
+|             |       |none  |     0|acc_norm|0.2201|±  |0.0121|
+|arc_easy     |      1|none  |     0|acc     |0.4373|±  |0.0102|
+|             |       |none  |     0|acc_norm|0.3935|±  |0.0100|
+|winogrande   |      1|none  |     0|acc     |0.5004|±  |0.0141|
+|openbookqa   |      1|none  |     0|acc     |0.1760|±  |0.0170|
+|             |       |none  |     0|acc_norm|0.2680|±  |0.0198|
+|hellaswag    |      1|none  |     0|acc     |0.2893|±  |0.0045|
+|             |       |none  |     0|acc_norm|0.3125|±  |0.0046|
+```
+
+You can use the following script to reproduce the results (assuming you have installed litgpt)
+```
+MODEL_NAME="step-00088000"
+MODEL_OUTPUT_ROOT="MicroLlamaV2-VastAI-Checkpoints/out/pretrain/micro-llama-v2"
+MODEL_OUTPUT_REL="${MODEL_OUTPUT_ROOT}/${MODEL_NAME}"
+
+# HuggingFace
+huggingface-cli download keeeeenw/MicroLlama2-checkpoints ${MODEL_NAME}/lit_model.pth --local-dir checkpoints/${MODEL_OUTPUT_ROOT}/
+huggingface-cli download keeeeenw/MicroLlama2-checkpoints ${MODEL_NAME}/generation_config.json --local-dir checkpoints/${MODEL_OUTPUT_ROOT}/
+huggingface-cli download keeeeenw/MicroLlama2-checkpoints ${MODEL_NAME}/hyperparameters.yaml --local-dir checkpoints/${MODEL_OUTPUT_ROOT}/
+huggingface-cli download keeeeenw/MicroLlama2-checkpoints ${MODEL_NAME}/model_config.yaml --local-dir checkpoints/${MODEL_OUTPUT_ROOT}/
+huggingface-cli download keeeeenw/MicroLlama2-checkpoints ${MODEL_NAME}/tokenizer.json --local-dir checkpoints/${MODEL_OUTPUT_ROOT}/
+huggingface-cli download keeeeenw/MicroLlama2-checkpoints ${MODEL_NAME}/tokenizer_config.json --local-dir checkpoints/${MODEL_OUTPUT_ROOT}/
+
+# Copy config, see "caveat" below
+cp -r <local_path>/config.json checkpoints/${MODEL_OUTPUT_REL}/
+
+# AWS
+# aws s3 cp s3://microllama-v2/checkpoints/out/pretrain/micro-llama-v2/${MODEL_NAME} checkpoints/${MODEL_OUTPUT_REL} --recursive
+
+litgpt evaluate \
+  ${MODEL_OUTPUT_REL} \
+  --tasks "hellaswag,openbookqa,winogrande,arc_easy,arc_challenge,boolq,piqa" \
+  --device cuda:0 \
+  --batch_size 16
+```
+**Caveat**: for some reason the auto generated config.json for the model in the checkpoint is incorrect, you will need to replace it with https://huggingface.co/keeeeenw/MicroLlama2-checkpoints/blob/main/config.json
+to resolve the evaluation error.
+
 # How to use these checkpoints
 
 These checkpoints are compatible with [litgpt](https://github.com/Lightning-AI/litgpt) with slight modifications (see section below).
@@ -40,6 +91,8 @@ Reference:
 1. litgpt pretrain checkpoint to inference checkpoint https://github.com/Lightning-AI/litgpt/blob/main/tutorials/pretrain_tinyllama.md#export-checkpoints
 2. litgpt inference checkpoint to HF checkpoints https://github.com/Lightning-AI/litgpt/blob/main/tutorials/convert_lit_models.md
 
+**Caveat**: for some reason the auto generated config.json for the model in the checkpoint is incorrect, you will need to replace it with https://huggingface.co/keeeeenw/MicroLlama2-checkpoints/blob/main/config.json
+to resolve any inference or evaluation error.
 
 # Advanced usage - pretraining with litgpt