Upload lora (adapter) -- gsm8k gate: PASS

README.md

@@ -1,80 +1,207 @@
 ---
-library_name: peft
 base_model: openbmb/MiniCPM5-1B
-license: apache-2.0
+library_name: peft
+pipeline_tag: text-generation
 tags:
-  - lora
-  - qlora
-  - build-small-hackathon
-  - well-tuned
-  - math
+- base_model:adapter:openbmb/MiniCPM5-1B
+- lora
+- transformers
 ---
 
-# math-lora
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
 
-QLoRA adapter for **math**, fine-tuned from `openbmb/MiniCPM5-1B` on `meta-math/MetaMathQA` + `tatsu-lab/alpaca` (format: `mix`).
+[More Information Needed]
 
-Trained, evaluated, and gated on [Modal](https://modal.com/docs/guide) via `research/modal/` (app `slm-finetune-benchmark`).
+#### Software
 
-## Benchmark gate
+[More Information Needed]
 
-- skill eval profile: `math`
-- gate: **PASSED**
+## Citation [optional]
 
-### Skill checks
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
 
-| check | value | result |
-| --- | ---: | --- |
-| gsm8k >= 0.05 | 0.4000 | pass |
-| gsm8k improve >= 0.02 | 0.0700 | pass |
-| arc_challenge regress <= 0.03 | -0.0500 | pass |
-| hellaswag regress <= 0.03 | 0.0000 | pass |
-| piqa regress <= 0.03 | 0.0200 | pass |
+**BibTeX:**
 
-- general eval profile: `compare_study`
+[More Information Needed]
 
-### General checks
+**APA:**
 
-| check | value | result |
-| --- | ---: | --- |
-| arc_easy regress <= 0.03 | -0.0300 | pass |
-| arc_challenge regress <= 0.03 | -0.0400 | pass |
-| hellaswag regress <= 0.03 | 0.0100 | pass |
-| piqa regress <= 0.03 | 0.0100 | pass |
-| boolq regress <= 0.03 | -0.0300 | pass |
-| gsm8k regress <= 0.03 | -0.0700 | pass |
+[More Information Needed]
 
+## Glossary [optional]
 
-## lm-eval results
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
 
-| task | metric | baseline | candidate | delta |
-| --- | --- | ---: | ---: | ---: |
-| arc_challenge | acc,none | 0.3200 | 0.3700 | +0.0500 |
-| gsm8k | exact_match,strict-match | 0.3300 | 0.4000 | +0.0700 |
-| hellaswag | acc,none | 0.4300 | 0.4300 | +0.0000 |
-| piqa | acc,none | 0.7200 | 0.7000 | -0.0200 |
+[More Information Needed]
 
-## Training
+## More Information [optional]
 
-- dataset: `/repo/research/data/education-lesson-chat.jsonl`
-- mode: `qlora`
-- samples: {'train': 3528, 'eval': 72}
-- final train loss: 0.340698
-- eval loss: 0.494981
+[More Information Needed]
 
-## Load with PEFT
+## Model Card Authors [optional]
 
-```python
-from peft import PeftModel
-from transformers import AutoModelForCausalLM, AutoTokenizer
+[More Information Needed]
 
-base = "openbmb/MiniCPM5-1B"
-adapter = "MSGEncrypted/minicpm5-1b-math-lora"
+## Model Card Contact
 
-tokenizer = AutoTokenizer.from_pretrained(base, trust_remote_code=True)
-model = AutoModelForCausalLM.from_pretrained(
-    base, torch_dtype="auto", device_map="auto", trust_remote_code=True
-)
-model = PeftModel.from_pretrained(model, adapter)
-```
+[More Information Needed]
+### Framework versions
 
+- PEFT 0.19.1

adapter_config.json

@@ -30,13 +30,13 @@
   "rank_pattern": {},
   "revision": null,
   "target_modules": [
+    "q_proj",
+    "up_proj",
+    "k_proj",
     "gate_proj",
-    "v_proj",
     "o_proj",
-    "k_proj",
-    "up_proj",
     "down_proj",
-    "q_proj"
+    "v_proj"
   ],
   "target_parameters": null,
   "task_type": "CAUSAL_LM",

adapter_model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e024be6b763d331e1aa3095616eb6b46419aeaa1736c27e3de299a627e035cda
+oid sha256:c05345873ae1104f37936489e46f8bb2698b09934726d06bb0fa3ee4fa978284
 size 89697856

checkpoint-120/README.md

@@ -0,0 +1,207 @@
+---
+base_model: openbmb/MiniCPM5-1B
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:openbmb/MiniCPM5-1B
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.19.1

checkpoint-120/adapter_config.json

@@ -0,0 +1,48 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "openbmb/MiniCPM5-1B",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 64,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 32,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "q_proj",
+    "up_proj",
+    "k_proj",
+    "gate_proj",
+    "o_proj",
+    "down_proj",
+    "v_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

checkpoint-120/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9feee39de6c602284f9c0100be03259062684e791f55dc80224a5e40d9a5d7e2
+size 89697856

checkpoint-120/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4dc6403bdfdbcc4670c4cf88b3eaf0fe674a066b5cabe599ff2fc5db59bcded7
+size 179594003

checkpoint-120/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:144151981637c94f01b8fda964b87f9c43760694c35e4f518f20ae67c6c08930
+size 14645

checkpoint-120/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:025329941a9b6b556fba96de6c32ce6e101e8ae4652aa090c274b3a5dc1cdfc2
+size 1465

checkpoint-120/trainer_state.json

@@ -0,0 +1,150 @@
+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.8421052631578947,
+  "eval_steps": 30,
+  "global_step": 120,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.07017543859649122,
+      "grad_norm": 0.4250744879245758,
+      "learning_rate": 0.00019962469712828614,
+      "loss": 0.6094220161437989,
+      "step": 10
+    },
+    {
+      "epoch": 0.14035087719298245,
+      "grad_norm": 0.30426689982414246,
+      "learning_rate": 0.00019543482507085482,
+      "loss": 0.5189157009124756,
+      "step": 20
+    },
+    {
+      "epoch": 0.21052631578947367,
+      "grad_norm": 0.3195377588272095,
+      "learning_rate": 0.00018678252666130013,
+      "loss": 0.4654271602630615,
+      "step": 30
+    },
+    {
+      "epoch": 0.21052631578947367,
+      "eval_loss": 0.4821685254573822,
+      "eval_runtime": 26.1957,
+      "eval_samples_per_second": 9.162,
+      "eval_steps_per_second": 2.29,
+      "step": 30
+    },
+    {
+      "epoch": 0.2807017543859649,
+      "grad_norm": 0.2729398012161255,
+      "learning_rate": 0.00017407237375691392,
+      "loss": 0.5165626049041748,
+      "step": 40
+    },
+    {
+      "epoch": 0.3508771929824561,
+      "grad_norm": 0.34597083926200867,
+      "learning_rate": 0.0001578986789811849,
+      "loss": 0.5297622203826904,
+      "step": 50
+    },
+    {
+      "epoch": 0.42105263157894735,
+      "grad_norm": 0.3952707350254059,
+      "learning_rate": 0.00013901770632605547,
+      "loss": 0.5074038982391358,
+      "step": 60
+    },
+    {
+      "epoch": 0.42105263157894735,
+      "eval_loss": 0.47542476654052734,
+      "eval_runtime": 25.7324,
+      "eval_samples_per_second": 9.327,
+      "eval_steps_per_second": 2.332,
+      "step": 60
+    },
+    {
+      "epoch": 0.49122807017543857,
+      "grad_norm": 0.2613320052623749,
+      "learning_rate": 0.00011831230908818563,
+      "loss": 0.5146247386932373,
+      "step": 70
+    },
+    {
+      "epoch": 0.5614035087719298,
+      "grad_norm": 0.2839556932449341,
+      "learning_rate": 9.675064863002196e-05,
+      "loss": 0.44298686981201174,
+      "step": 80
+    },
+    {
+      "epoch": 0.631578947368421,
+      "grad_norm": 0.3424423635005951,
+      "learning_rate": 7.534092423052381e-05,
+      "loss": 0.44837327003479005,
+      "step": 90
+    },
+    {
+      "epoch": 0.631578947368421,
+      "eval_loss": 0.472128301858902,
+      "eval_runtime": 25.8951,
+      "eval_samples_per_second": 9.268,
+      "eval_steps_per_second": 2.317,
+      "step": 90
+    },
+    {
+      "epoch": 0.7017543859649122,
+      "grad_norm": 0.2950887680053711,
+      "learning_rate": 5.5084230807412126e-05,
+      "loss": 0.4572721481323242,
+      "step": 100
+    },
+    {
+      "epoch": 0.7719298245614035,
+      "grad_norm": 0.3997437059879303,
+      "learning_rate": 3.6927748831453836e-05,
+      "loss": 0.4888507843017578,
+      "step": 110
+    },
+    {
+      "epoch": 0.8421052631578947,
+      "grad_norm": 0.30841732025146484,
+      "learning_rate": 2.1720455220364444e-05,
+      "loss": 0.4997568130493164,
+      "step": 120
+    },
+    {
+      "epoch": 0.8421052631578947,
+      "eval_loss": 0.46787339448928833,
+      "eval_runtime": 25.8471,
+      "eval_samples_per_second": 9.285,
+      "eval_steps_per_second": 2.321,
+      "step": 120
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 150,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 2,
+  "save_steps": 30,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 7855484285214720.0,
+  "train_batch_size": 4,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-120/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f472359513759eb48389708123f1d100f299bb3edd283fefda43dd315dad87b8
+size 5201

checkpoint-150/adapter_config.json

@@ -30,13 +30,13 @@
   "rank_pattern": {},
   "revision": null,
   "target_modules": [
+    "q_proj",
+    "up_proj",
+    "k_proj",
     "gate_proj",
-    "v_proj",
     "o_proj",
-    "k_proj",
-    "up_proj",
     "down_proj",
-    "q_proj"
+    "v_proj"
   ],
   "target_parameters": null,
   "task_type": "CAUSAL_LM",

checkpoint-150/adapter_model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e024be6b763d331e1aa3095616eb6b46419aeaa1736c27e3de299a627e035cda
+oid sha256:c05345873ae1104f37936489e46f8bb2698b09934726d06bb0fa3ee4fa978284
 size 89697856

checkpoint-150/optimizer.pt

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:da736af699c86703b742434ea40618d38ffa85320a34d95c1bbcb7e5da923310
+oid sha256:25c29a1df9450f4dc58cd000bfdf8cd805efabf18e0ecf73b0397684102309cb
 size 179594003

checkpoint-150/rng_state.pth

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8077321b6911087294dc7b05c9a860579d6291a34cbd5abc1975ab53b3a0b0f3
+oid sha256:ac04c24a7525782166d7cb5b96826a04b081fbe4e3f0b916529585df9c36a42a
 size 14645

checkpoint-150/trainer_state.json

@@ -2,7 +2,7 @@
   "best_global_step": null,
   "best_metric": null,
   "best_model_checkpoint": null,
-  "epoch": 0.6802721088435374,
+  "epoch": 1.0491228070175438,
   "eval_steps": 30,
   "global_step": 150,
   "is_hyper_param_search": false,
@@ -10,156 +10,156 @@
   "is_world_process_zero": true,
   "log_history": [
     {
-      "epoch": 0.045351473922902494,
-      "grad_norm": 0.5955151915550232,
+      "epoch": 0.07017543859649122,
+      "grad_norm": 0.4250744879245758,
       "learning_rate": 0.00019962469712828614,
-      "loss": 0.4111457824707031,
+      "loss": 0.6094220161437989,
       "step": 10
     },
     {
-      "epoch": 0.09070294784580499,
-      "grad_norm": 0.5896233320236206,
+      "epoch": 0.14035087719298245,
+      "grad_norm": 0.30426689982414246,
       "learning_rate": 0.00019543482507085482,
-      "loss": 0.35993859767913816,
+      "loss": 0.5189157009124756,
       "step": 20
     },
     {
-      "epoch": 0.1360544217687075,
-      "grad_norm": 0.48774707317352295,
+      "epoch": 0.21052631578947367,
+      "grad_norm": 0.3195377588272095,
       "learning_rate": 0.00018678252666130013,
-      "loss": 0.38935122489929197,
+      "loss": 0.4654271602630615,
       "step": 30
     },
     {
-      "epoch": 0.1360544217687075,
-      "eval_loss": 0.5217297077178955,
-      "eval_runtime": 2.0775,
-      "eval_samples_per_second": 34.657,
-      "eval_steps_per_second": 8.664,
+      "epoch": 0.21052631578947367,
+      "eval_loss": 0.4821685254573822,
+      "eval_runtime": 26.1957,
+      "eval_samples_per_second": 9.162,
+      "eval_steps_per_second": 2.29,
       "step": 30
     },
     {
-      "epoch": 0.18140589569160998,
-      "grad_norm": 0.4432049095630646,
+      "epoch": 0.2807017543859649,
+      "grad_norm": 0.2729398012161255,
       "learning_rate": 0.00017407237375691392,
-      "loss": 0.3316951274871826,
+      "loss": 0.5165626049041748,
       "step": 40
     },
     {
-      "epoch": 0.22675736961451248,
-      "grad_norm": 0.40836411714553833,
+      "epoch": 0.3508771929824561,
+      "grad_norm": 0.34597083926200867,
       "learning_rate": 0.0001578986789811849,
-      "loss": 0.31510772705078127,
+      "loss": 0.5297622203826904,
       "step": 50
     },
     {
-      "epoch": 0.272108843537415,
-      "grad_norm": 0.3234673738479614,
+      "epoch": 0.42105263157894735,
+      "grad_norm": 0.3952707350254059,
       "learning_rate": 0.00013901770632605547,
-      "loss": 0.3653845310211182,
+      "loss": 0.5074038982391358,
       "step": 60
     },
     {
-      "epoch": 0.272108843537415,
-      "eval_loss": 0.5070984363555908,
-      "eval_runtime": 2.0635,
-      "eval_samples_per_second": 34.891,
-      "eval_steps_per_second": 8.723,
+      "epoch": 0.42105263157894735,
+      "eval_loss": 0.47542476654052734,
+      "eval_runtime": 25.7324,
+      "eval_samples_per_second": 9.327,
+      "eval_steps_per_second": 2.332,
       "step": 60
     },
     {
-      "epoch": 0.31746031746031744,
-      "grad_norm": 0.28090140223503113,
+      "epoch": 0.49122807017543857,
+      "grad_norm": 0.2613320052623749,
       "learning_rate": 0.00011831230908818563,
-      "loss": 0.2907076835632324,
+      "loss": 0.5146247386932373,
       "step": 70
     },
     {
-      "epoch": 0.36281179138321995,
-      "grad_norm": 0.33969420194625854,
+      "epoch": 0.5614035087719298,
+      "grad_norm": 0.2839556932449341,
       "learning_rate": 9.675064863002196e-05,
-      "loss": 0.3239952325820923,
+      "loss": 0.44298686981201174,
       "step": 80
     },
     {
-      "epoch": 0.40816326530612246,
-      "grad_norm": 0.3883151113986969,
+      "epoch": 0.631578947368421,
+      "grad_norm": 0.3424423635005951,
       "learning_rate": 7.534092423052381e-05,
-      "loss": 0.39945359230041505,
+      "loss": 0.44837327003479005,
       "step": 90
     },
     {
-      "epoch": 0.40816326530612246,
-      "eval_loss": 0.49975717067718506,
-      "eval_runtime": 2.0811,
-      "eval_samples_per_second": 34.597,
-      "eval_steps_per_second": 8.649,
+      "epoch": 0.631578947368421,
+      "eval_loss": 0.472128301858902,
+      "eval_runtime": 25.8951,
+      "eval_samples_per_second": 9.268,
+      "eval_steps_per_second": 2.317,
       "step": 90
     },
     {
-      "epoch": 0.45351473922902497,
-      "grad_norm": 0.37783923745155334,
+      "epoch": 0.7017543859649122,
+      "grad_norm": 0.2950887680053711,
       "learning_rate": 5.5084230807412126e-05,
-      "loss": 0.33277587890625,
+      "loss": 0.4572721481323242,
       "step": 100
     },
     {
-      "epoch": 0.4988662131519274,
-      "grad_norm": 0.44748321175575256,
+      "epoch": 0.7719298245614035,
+      "grad_norm": 0.3997437059879303,
       "learning_rate": 3.6927748831453836e-05,
-      "loss": 0.3025733470916748,
+      "loss": 0.4888507843017578,
       "step": 110
     },
     {
-      "epoch": 0.54421768707483,
-      "grad_norm": 0.39024776220321655,
+      "epoch": 0.8421052631578947,
+      "grad_norm": 0.30841732025146484,
       "learning_rate": 2.1720455220364444e-05,
-      "loss": 0.30908067226409913,
+      "loss": 0.4997568130493164,
       "step": 120
     },
     {
-      "epoch": 0.54421768707483,
-      "eval_loss": 0.4952836036682129,
-      "eval_runtime": 2.0845,
-      "eval_samples_per_second": 34.541,
-      "eval_steps_per_second": 8.635,
+      "epoch": 0.8421052631578947,
+      "eval_loss": 0.46787339448928833,
+      "eval_runtime": 25.8471,
+      "eval_samples_per_second": 9.285,
+      "eval_steps_per_second": 2.321,
       "step": 120
     },
     {
-      "epoch": 0.5895691609977324,
-      "grad_norm": 0.39623892307281494,
+      "epoch": 0.9122807017543859,
+      "grad_norm": 0.31729358434677124,
       "learning_rate": 1.0173426121705576e-05,
-      "loss": 0.3232832193374634,
+      "loss": 0.45314512252807615,
       "step": 130
     },
     {
-      "epoch": 0.6349206349206349,
-      "grad_norm": 0.3841199576854706,
+      "epoch": 0.9824561403508771,
+      "grad_norm": 0.3034511208534241,
       "learning_rate": 2.826587782529444e-06,
-      "loss": 0.3398961782455444,
+      "loss": 0.41479806900024413,
       "step": 140
     },
     {
-      "epoch": 0.6802721088435374,
-      "grad_norm": 0.4336070120334625,
+      "epoch": 1.0491228070175438,
+      "grad_norm": 0.3052826523780823,
       "learning_rate": 2.347019815158724e-08,
-      "loss": 0.34069836139678955,
+      "loss": 0.45969629287719727,
       "step": 150
     },
     {
-      "epoch": 0.6802721088435374,
-      "eval_loss": 0.49498093128204346,
-      "eval_runtime": 2.0755,
-      "eval_samples_per_second": 34.691,
-      "eval_steps_per_second": 8.673,
+      "epoch": 1.0491228070175438,
+      "eval_loss": 0.46667739748954773,
+      "eval_runtime": 25.8488,
+      "eval_samples_per_second": 9.285,
+      "eval_steps_per_second": 2.321,
       "step": 150
     }
   ],
   "logging_steps": 10,
   "max_steps": 150,
   "num_input_tokens_seen": 0,
-  "num_train_epochs": 1,
-  "save_steps": 500,
+  "num_train_epochs": 2,
+  "save_steps": 30,
   "stateful_callbacks": {
     "TrainerControl": {
       "args": {
@@ -172,7 +172,7 @@
       "attributes": {}
     }
   },
-  "total_flos": 4361616479379456.0,
+  "total_flos": 9766840338321408.0,
   "train_batch_size": 4,
   "trial_name": null,
   "trial_params": null

checkpoint-150/training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c132835da72e5a98a55da309abaadcb858dfcd4c53b6c7dfaf23d45de30ee7a7
+oid sha256:f472359513759eb48389708123f1d100f299bb3edd283fefda43dd315dad87b8
 size 5201

eval_results.json

@@ -0,0 +1,34 @@
+{
+  "gsm8k": {
+    "before": {
+      "n": 30,
+      "correct": 4,
+      "accuracy": 0.13333333333333333,
+      "elapsed_sec": 235.86863923072815,
+      "seed": 42
+    },
+    "after": {
+      "n": 30,
+      "correct": 6,
+      "accuracy": 0.2,
+      "elapsed_sec": 241.18271040916443,
+      "seed": 42
+    }
+  },
+  "piqa": {
+    "before": {
+      "n": 150,
+      "correct": 93,
+      "accuracy": 0.62,
+      "elapsed_sec": 13.179616451263428,
+      "seed": 42
+    },
+    "after": {
+      "n": 150,
+      "correct": 97,
+      "accuracy": 0.6466666666666666,
+      "elapsed_sec": 23.426591634750366,
+      "seed": 42
+    }
+  }
+}

tokenizer_config.json

@@ -6,8 +6,8 @@
   "eos_token": "</s>",
   "is_local": false,
   "legacy": true,
-  "local_files_only": false,
   "model_max_length": 1000000000000000019884624838656,
+  "model_specific_special_tokens": {},
   "pad_token": "</s>",
   "sp_model_kwargs": {},
   "spaces_between_special_tokens": false,

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c132835da72e5a98a55da309abaadcb858dfcd4c53b6c7dfaf23d45de30ee7a7
+oid sha256:f472359513759eb48389708123f1d100f299bb3edd283fefda43dd315dad87b8
 size 5201

training_manifest.json

@@ -0,0 +1,44 @@
+{
+  "saved_at": "2026-07-17T14:31:31.300831+00:00",
+  "mode": "lora",
+  "base_model": "openbmb/MiniCPM5-1B",
+  "model_key": "minicpm5-1b",
+  "datasets": [
+    {
+      "key": "meta_math",
+      "path": "meta-math/MetaMathQA",
+      "split": "train[:3000]",
+      "config": null
+    },
+    {
+      "key": "hendrycks_math_algebra",
+      "path": "EleutherAI/hendrycks_math",
+      "split": "train",
+      "config": "algebra"
+    },
+    {
+      "key": "hendrycks_math_geometry",
+      "path": "EleutherAI/hendrycks_math",
+      "split": "train",
+      "config": "geometry"
+    }
+  ],
+  "training": {
+    "epochs": 1.0,
+    "max_steps": 150,
+    "batch_size": 2,
+    "grad_accum": 8,
+    "max_len": 512,
+    "lora_r": 32,
+    "lora_alpha": 64,
+    "neftune_noise_alpha": 5.0
+  },
+  "metrics": {
+    "train_loss": 0.48846651395161944,
+    "eval_loss": 0.46667739748954773,
+    "perplexity": 1.5946868704388588
+  },
+  "load_hint": {
+    "adapter": "PeftModel.from_pretrained(base, \"./minicpm5-1b-math-lora\")"
+  }
+}

training_results.json

@@ -1,46 +1,5 @@
 {
-  "model": "openbmb/MiniCPM5-1B",
-  "preset": "minicpm5-1b",
-  "dataset": "/repo/research/data/education-lesson-chat.jsonl",
-  "dataset_config": null,
-  "dataset_split": "train",
-  "mix": [
-    {
-      "dataset": "meta-math/MetaMathQA",
-      "format": "prompt",
-      "columns": {
-        "prompt": "query",
-        "response": "response"
-      },
-      "dataset_split": "train[:3000]",
-      "max_samples": 3000
-    },
-    {
-      "dataset": "tatsu-lab/alpaca",
-      "format": "alpaca",
-      "dataset_split": "train[:600]",
-      "max_samples": 600
-    }
-  ],
-  "format": "chat",
-  "mode": "qlora",
-  "output_dir": "/vol/finetuned/math-lora",
-  "samples": {
-    "train": 3528,
-    "eval": 72
-  },
-  "metrics": {
-    "final_train_loss": 0.340698,
-    "eval_loss": 0.494981,
-    "perplexity": 1.6405,
-    "loss_score": 0.494981,
-    "result_score": 60.96
-  },
-  "training": {
-    "epochs": 1.0,
-    "max_steps": 150,
-    "global_step": 150,
-    "train_runtime_sec": 310.98,
-    "train_samples_per_second": 7.718
-  }
+  "train_loss": 0.48846651395161944,
+  "eval_loss": 0.46667739748954773,
+  "perplexity": 1.5946868704388588
 }