Upload folder using huggingface_hub
Browse files- sld_26.0/panel_counter_v20_27b_sft/README.md +29 -0
- sld_26.0/panel_counter_v20_27b_sft/eval/phase0_serve_stop.log +4 -0
- sld_26.0/panel_counter_v20_27b_sft/eval/phase2_sft.log +0 -0
- sld_26.0/panel_counter_v20_27b_sft/eval/phase3_smoke.log +0 -0
- sld_26.0/panel_counter_v20_27b_sft/eval/phase4_grpo_20260705-155119.log +0 -0
- sld_26.0/panel_counter_v20_27b_sft/eval/run_20260704-161913.log +108 -0
- sld_26.0/panel_counter_v20_27b_sft/eval/run_20260704-162424.log +0 -0
- sld_26.0/panel_counter_v20_27b_sft/eval/run_20260705-150056.log +0 -0
- sld_26.0/panel_counter_v20_27b_sft/eval/v20_27b_sft_best_ckpt.txt +1 -0
- sld_26.0/panel_counter_v20_27b_sft/training_logging_full.jsonl +27 -0
sld_26.0/panel_counter_v20_27b_sft/README.md
ADDED
|
@@ -0,0 +1,29 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# panel_counter_v20_27b_sft (archived, superseded)
|
| 2 |
+
|
| 3 |
+
Source: `sld_26.0/training/checkpoints/panel_counter_v20_27b_sft`
|
| 4 |
+
Final checkpoint archived: `checkpoint-32` (step 32) from run `v0-20260704-162435`
|
| 5 |
+
Training run segments (resumed sequentially): ['v0-20260704-162435']
|
| 6 |
+
|
| 7 |
+
This is an **inference-only** archive: adapter weights + configs + full training log +
|
| 8 |
+
eval results. Optimizer/scheduler/RNG state (needed only to resume this exact run) was
|
| 9 |
+
dropped to save space; this run is superseded by later versions and not intended to be
|
| 10 |
+
resumed.
|
| 11 |
+
|
| 12 |
+
## Training summary
|
| 13 |
+
```
|
| 14 |
+
{
|
| 15 |
+
"loss_first": 0.69220698,
|
| 16 |
+
"loss_last": 0.0113448
|
| 17 |
+
}
|
| 18 |
+
```
|
| 19 |
+
Total logged steps across all resumed segments: 27
|
| 20 |
+
|
| 21 |
+
## Eval files included
|
| 22 |
+
- eval/phase0_serve_stop.log
|
| 23 |
+
- eval/phase2_sft.log
|
| 24 |
+
- eval/phase3_smoke.log
|
| 25 |
+
- eval/phase4_grpo_20260705-155119.log
|
| 26 |
+
- eval/run_20260704-161913.log
|
| 27 |
+
- eval/run_20260704-162424.log
|
| 28 |
+
- eval/run_20260705-150056.log
|
| 29 |
+
- eval/v20_27b_sft_best_ckpt.txt
|
sld_26.0/panel_counter_v20_27b_sft/eval/phase0_serve_stop.log
ADDED
|
@@ -0,0 +1,4 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
|
| 2 |
+
[serve_v19] target gpu-memory-utilization=stop extra_lora=none
|
| 3 |
+
[serve_v19] no existing vLLM process found
|
| 4 |
+
[serve_v19] stop requested — vLLM killed, not relaunching
|
sld_26.0/panel_counter_v20_27b_sft/eval/phase2_sft.log
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
sld_26.0/panel_counter_v20_27b_sft/eval/phase3_smoke.log
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
sld_26.0/panel_counter_v20_27b_sft/eval/phase4_grpo_20260705-155119.log
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
sld_26.0/panel_counter_v20_27b_sft/eval/run_20260704-161913.log
ADDED
|
@@ -0,0 +1,108 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
|
| 2 |
+
[phase 0] preflight — 2026-07-04T16:19:13+00:00
|
| 3 |
+
33886 MiB, 11573 MiB, 46068 MiB
|
| 4 |
+
/dev/nvme1n1 850G 679G 171G 80% /workspace
|
| 5 |
+
bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
|
| 6 |
+
[serve_v19] target gpu-memory-utilization=stop extra_lora=none
|
| 7 |
+
[serve_v19] killing existing vLLM pids: 1098035
|
| 8 |
+
1098138
|
| 9 |
+
[serve_v19] SIGKILL fallback for pid 1098035
|
| 10 |
+
[serve_v19] stop requested — vLLM killed, not relaunching
|
| 11 |
+
[phase 0] done at 3s
|
| 12 |
+
[phase 1] data check — 2026-07-04T16:19:16+00:00
|
| 13 |
+
reusing: sft_v19_pure30_train.jsonl (27), sft_v19_pure30_eval.jsonl (3), grpo_v19_pure30_train.jsonl (27)
|
| 14 |
+
[phase 1] done at 3s
|
| 15 |
+
[phase 2] QLoRA SFT (Qwen3.6-27B) — 2026-07-04T16:19:16+00:00
|
| 16 |
+
bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
|
| 17 |
+
run sh: `/usr/bin/python3 /home/node/.local/lib/python3.11/site-packages/swift/cli/sft.py --model Qwen/Qwen3.6-27B --model_type qwen3_5 --template qwen3_5 --quant_method bnb --quant_bits 4 --tuner_type lora --dataset /workspace/training/sft_v19_pure30_train.jsonl --val_dataset /workspace/training/sft_v19_pure30_eval.jsonl --output_dir /workspace/training/checkpoints/panel_counter_v20_27b_sft --torch_dtype bfloat16 --lora_rank 64 --lora_alpha 128 --target_modules all-linear --num_train_epochs 15 --per_device_train_batch_size 1 --per_device_eval_batch_size 1 --gradient_accumulation_steps 8 --learning_rate 5e-5 --warmup_ratio 0.05 --weight_decay 0.1 --lr_scheduler_type cosine --max_length 8192 --eval_strategy steps --eval_steps 4 --save_strategy steps --save_steps 4 --save_total_limit 4 --load_best_model_at_end true --metric_for_best_model loss --greater_is_better false --early_stop_interval 3 --logging_steps 2 --bf16 true --gradient_checkpointing true --optim adamw_torch_fused --enable_thinking false --report_to none`
|
| 18 |
+
[INFO:swift] Successfully registered `/home/node/.local/lib/python3.11/site-packages/swift/dataset/data/dataset_info.json`.
|
| 19 |
+
[INFO:swift] rank: -1, local_rank: -1, world_size: 1, local_world_size: 1
|
| 20 |
+
[INFO:swift] Downloading the model from HuggingFace Hub, model_id: Qwen/Qwen3.6-27B
|
| 21 |
+
|
| 22 |
+
|
| 23 |
+
|
| 24 |
+
Traceback (most recent call last):
|
| 25 |
+
File "/home/node/.local/lib/python3.11/site-packages/swift/cli/sft.py", line 20, in <module>
|
| 26 |
+
sft_main()
|
| 27 |
+
File "/home/node/.local/lib/python3.11/site-packages/swift/pipelines/train/sft.py", line 354, in sft_main
|
| 28 |
+
return SwiftSft(args).main()
|
| 29 |
+
^^^^^^^^^^^^^^
|
| 30 |
+
File "/home/node/.local/lib/python3.11/site-packages/swift/pipelines/train/sft.py", line 27, in __init__
|
| 31 |
+
super().__init__(args)
|
| 32 |
+
File "/home/node/.local/lib/python3.11/site-packages/swift/pipelines/base.py", line 18, in __init__
|
| 33 |
+
self.args = self._parse_args(args)
|
| 34 |
+
^^^^^^^^^^^^^^^^^^^^^^
|
| 35 |
+
File "/home/node/.local/lib/python3.11/site-packages/swift/pipelines/base.py", line 35, in _parse_args
|
| 36 |
+
args, remaining_argv = parse_args(self.args_class, args)
|
| 37 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 38 |
+
File "/home/node/.local/lib/python3.11/site-packages/swift/utils/utils.py", line 182, in parse_args
|
| 39 |
+
args, remaining_args = parser.parse_args_into_dataclasses(argv, return_remaining_strings=True)
|
| 40 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 41 |
+
File "/home/node/.local/lib/python3.11/site-packages/transformers/hf_argparser.py", line 345, in parse_args_into_dataclasses
|
| 42 |
+
obj = dtype(**inputs)
|
| 43 |
+
^^^^^^^^^^^^^^^
|
| 44 |
+
File "<string>", line 320, in __init__
|
| 45 |
+
File "/home/node/.local/lib/python3.11/site-packages/swift/arguments/sft_args.py", line 204, in __post_init__
|
| 46 |
+
BaseArguments.__post_init__(self)
|
| 47 |
+
File "/home/node/.local/lib/python3.11/site-packages/swift/arguments/base_args/base_args.py", line 167, in __post_init__
|
| 48 |
+
ModelArguments.__post_init__(self)
|
| 49 |
+
File "/home/node/.local/lib/python3.11/site-packages/swift/arguments/base_args/model_args.py", line 224, in __post_init__
|
| 50 |
+
self._init_torch_dtype()
|
| 51 |
+
File "/home/node/.local/lib/python3.11/site-packages/swift/arguments/base_args/model_args.py", line 123, in _init_torch_dtype
|
| 52 |
+
self.torch_dtype: torch.dtype = self._init_model_info()
|
| 53 |
+
^^^^^^^^^^^^^^^^^^^^^^^
|
| 54 |
+
File "/home/node/.local/lib/python3.11/site-packages/swift/arguments/base_args/model_args.py", line 193, in _init_model_info
|
| 55 |
+
self.model_info, self.model_meta = get_model_info_meta(**model_kwargs)
|
| 56 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 57 |
+
File "/home/node/.local/lib/python3.11/site-packages/swift/model/model_meta.py", line 265, in get_model_info_meta
|
| 58 |
+
model_dir = safe_snapshot_download(
|
| 59 |
+
^^^^^^^^^^^^^^^^^^^^^^^
|
| 60 |
+
File "/home/node/.local/lib/python3.11/site-packages/swift/utils/hub_utils.py", line 100, in safe_snapshot_download
|
| 61 |
+
model_dir = hub.download_model(model_id_or_path, revision, ignore_patterns, token=hub_token, **kwargs)
|
| 62 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 63 |
+
File "/home/node/.local/lib/python3.11/site-packages/swift/hub/hub.py", line 455, in download_model
|
| 64 |
+
return snapshot_download(
|
| 65 |
+
^^^^^^^^^^^^^^^^^^
|
| 66 |
+
File "/home/node/.local/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 88, in _inner_fn
|
| 67 |
+
return fn(*args, **kwargs)
|
| 68 |
+
^^^^^^^^^^^^^^^^^^^
|
| 69 |
+
File "/home/node/.local/lib/python3.11/site-packages/huggingface_hub/_snapshot_download.py", line 455, in snapshot_download
|
| 70 |
+
thread_map(
|
| 71 |
+
File "/home/node/.local/lib/python3.11/site-packages/tqdm/contrib/concurrent.py", line 69, in thread_map
|
| 72 |
+
return _executor_map(ThreadPoolExecutor, fn, *iterables, **tqdm_kwargs)
|
| 73 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 74 |
+
File "/home/node/.local/lib/python3.11/site-packages/tqdm/contrib/concurrent.py", line 51, in _executor_map
|
| 75 |
+
return list(tqdm_class(ex.map(fn, *iterables, chunksize=chunksize), **kwargs))
|
| 76 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 77 |
+
File "/home/node/.local/lib/python3.11/site-packages/tqdm/std.py", line 1181, in __iter__
|
| 78 |
+
for obj in iterable:
|
| 79 |
+
File "/usr/lib/python3.11/concurrent/futures/_base.py", line 619, in result_iterator
|
| 80 |
+
yield _result_or_cancel(fs.pop())
|
| 81 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 82 |
+
File "/usr/lib/python3.11/concurrent/futures/_base.py", line 317, in _result_or_cancel
|
| 83 |
+
return fut.result(timeout)
|
| 84 |
+
^^^^^^^^^^^^^^^^^^^
|
| 85 |
+
File "/usr/lib/python3.11/concurrent/futures/_base.py", line 456, in result
|
| 86 |
+
return self.__get_result()
|
| 87 |
+
^^^^^^^^^^^^^^^^^^^
|
| 88 |
+
File "/usr/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
|
| 89 |
+
raise self._exception
|
| 90 |
+
File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
|
| 91 |
+
result = self.fn(*self.args, **self.kwargs)
|
| 92 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 93 |
+
File "/home/node/.local/lib/python3.11/site-packages/huggingface_hub/_snapshot_download.py", line 435, in _inner_hf_hub_download
|
| 94 |
+
hf_hub_download( # type: ignore
|
| 95 |
+
File "/home/node/.local/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 88, in _inner_fn
|
| 96 |
+
return fn(*args, **kwargs)
|
| 97 |
+
^^^^^^^^^^^^^^^^^^^
|
| 98 |
+
File "/home/node/.local/lib/python3.11/site-packages/huggingface_hub/file_download.py", line 997, in hf_hub_download
|
| 99 |
+
return _hf_hub_download_to_cache_dir(
|
| 100 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 101 |
+
File "/home/node/.local/lib/python3.11/site-packages/huggingface_hub/file_download.py", line 1216, in _hf_hub_download_to_cache_dir
|
| 102 |
+
_download_to_tmp_and_move(
|
| 103 |
+
File "/home/node/.local/lib/python3.11/site-packages/huggingface_hub/file_download.py", line 1842, in _download_to_tmp_and_move
|
| 104 |
+
xet_get(
|
| 105 |
+
File "/home/node/.local/lib/python3.11/site-packages/huggingface_hub/file_download.py", line 558, in xet_get
|
| 106 |
+
download_files(
|
| 107 |
+
RuntimeError: Data processing error: I/O error: Permission denied (os error 13)
|
| 108 |
+
FATAL: train_v20_27b_sft.sh failed (exit 1) — check phase2_sft.log (likely model download, template mismatch, or OOM)
|
sld_26.0/panel_counter_v20_27b_sft/eval/run_20260704-162424.log
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
sld_26.0/panel_counter_v20_27b_sft/eval/run_20260705-150056.log
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
sld_26.0/panel_counter_v20_27b_sft/eval/v20_27b_sft_best_ckpt.txt
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
/workspace/training/checkpoints/panel_counter_v20_27b_sft/v0-20260704-162435/checkpoint-20
|
sld_26.0/panel_counter_v20_27b_sft/training_logging_full.jsonl
ADDED
|
@@ -0,0 +1,27 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{"loss": 0.69220698, "grad_norm": 1.64300931, "learning_rate": 1.667e-05, "token_acc": 0.84332458, "epoch": 0.2962963, "global_step/max_steps": "1/60", "elapsed_time": "59s", "remaining_time": "57m 47s", "memory(GiB)": 31.78, "train_speed(s/it)": 58.756978}
|
| 2 |
+
{"loss": 0.57712334, "grad_norm": 1.53478515, "learning_rate": 3.333e-05, "token_acc": 0.86485062, "epoch": 0.59259259, "global_step/max_steps": "2/60", "elapsed_time": "2m 4s", "remaining_time": "59m 52s", "memory(GiB)": 36.88, "train_speed(s/it)": 61.923209}
|
| 3 |
+
{"loss": 0.56144077, "grad_norm": 1.18218708, "learning_rate": 4.996e-05, "token_acc": 0.85854752, "epoch": 1.0, "global_step/max_steps": "4/60", "elapsed_time": "3m 23s", "remaining_time": "47m 18s", "memory(GiB)": 36.88, "train_speed(s/it)": 50.671324}
|
| 4 |
+
{"eval_loss": 0.41863182, "eval_runtime": 6.1787, "eval_samples_per_second": 0.486, "eval_steps_per_second": 0.486, "eval_token_acc": 0.89798206, "epoch": 1.0, "global_step/max_steps": "4/60", "elapsed_time": "3m 29s", "remaining_time": "48m 44s", "memory(GiB)": 36.88, "train_speed(s/it)": 52.216365}
|
| 5 |
+
{"loss": 0.37389919, "grad_norm": 1.41246355, "learning_rate": 4.966e-05, "token_acc": 0.90602837, "epoch": 1.59259259, "global_step/max_steps": "6/60", "elapsed_time": "5m 37s", "remaining_time": "50m 31s", "memory(GiB)": 36.88, "train_speed(s/it)": 56.133988}
|
| 6 |
+
{"loss": 0.20407102, "grad_norm": 1.15488136, "learning_rate": 4.906e-05, "token_acc": 0.93345383, "epoch": 2.0, "global_step/max_steps": "8/60", "elapsed_time": "6m 58s", "remaining_time": "45m 16s", "memory(GiB)": 36.88, "train_speed(s/it)": 52.223393}
|
| 7 |
+
{"eval_loss": 0.14844364, "eval_runtime": 6.1727, "eval_samples_per_second": 0.486, "eval_steps_per_second": 0.486, "eval_token_acc": 0.95179372, "epoch": 2.0, "global_step/max_steps": "8/60", "elapsed_time": "7m 4s", "remaining_time": "45m 56s", "memory(GiB)": 36.88, "train_speed(s/it)": 52.995161}
|
| 8 |
+
{"loss": 0.13367745, "grad_norm": 0.88236737, "learning_rate": 4.816e-05, "token_acc": 0.96167704, "epoch": 2.59259259, "global_step/max_steps": "10/60", "elapsed_time": "9m 7s", "remaining_time": "45m 35s", "memory(GiB)": 36.88, "train_speed(s/it)": 54.693103}
|
| 9 |
+
{"loss": 0.12161177, "grad_norm": 0.55588925, "learning_rate": 4.699e-05, "token_acc": 0.95886416, "epoch": 3.0, "global_step/max_steps": "12/60", "elapsed_time": "10m 31s", "remaining_time": "42m 3s", "memory(GiB)": 36.88, "train_speed(s/it)": 52.571332}
|
| 10 |
+
{"eval_loss": 0.0944334, "eval_runtime": 6.19, "eval_samples_per_second": 0.485, "eval_steps_per_second": 0.485, "eval_token_acc": 0.9646861, "epoch": 3.0, "global_step/max_steps": "12/60", "elapsed_time": "10m 37s", "remaining_time": "42m 28s", "memory(GiB)": 36.88, "train_speed(s/it)": 53.08729}
|
| 11 |
+
{"loss": 0.07671892, "grad_norm": 0.42686018, "learning_rate": 4.554e-05, "token_acc": 0.97608539, "epoch": 3.59259259, "global_step/max_steps": "14/60", "elapsed_time": "12m 42s", "remaining_time": "41m 44s", "memory(GiB)": 36.88, "train_speed(s/it)": 54.437269}
|
| 12 |
+
{"loss": 0.07431428, "grad_norm": 0.73220539, "learning_rate": 4.385e-05, "token_acc": 0.97639521, "epoch": 4.0, "global_step/max_steps": "16/60", "elapsed_time": "14m 3s", "remaining_time": "38m 40s", "memory(GiB)": 36.88, "train_speed(s/it)": 52.716152}
|
| 13 |
+
{"eval_loss": 0.08191577, "eval_runtime": 6.1494, "eval_samples_per_second": 0.488, "eval_steps_per_second": 0.488, "eval_token_acc": 0.97141256, "epoch": 4.0, "global_step/max_steps": "16/60", "elapsed_time": "14m 10s", "remaining_time": "38m 56s", "memory(GiB)": 36.88, "train_speed(s/it)": 53.100578}
|
| 14 |
+
{"loss": 0.04453896, "grad_norm": 0.36494333, "learning_rate": 4.193e-05, "token_acc": 0.98697663, "epoch": 4.59259259, "global_step/max_steps": "18/60", "elapsed_time": "16m 8s", "remaining_time": "37m 38s", "memory(GiB)": 36.88, "train_speed(s/it)": 53.753852}
|
| 15 |
+
{"loss": 0.05436675, "grad_norm": 0.53516477, "learning_rate": 3.981e-05, "token_acc": 0.98176455, "epoch": 5.0, "global_step/max_steps": "20/60", "elapsed_time": "17m 36s", "remaining_time": "35m 11s", "memory(GiB)": 36.88, "train_speed(s/it)": 52.780275}
|
| 16 |
+
{"eval_loss": 0.07116517, "eval_runtime": 6.182, "eval_samples_per_second": 0.485, "eval_steps_per_second": 0.485, "eval_token_acc": 0.97757848, "epoch": 5.0, "global_step/max_steps": "20/60", "elapsed_time": "17m 42s", "remaining_time": "35m 24s", "memory(GiB)": 36.88, "train_speed(s/it)": 53.089451}
|
| 17 |
+
{"loss": 0.04044443, "grad_norm": 0.38481435, "learning_rate": 3.75e-05, "token_acc": 0.98828402, "epoch": 5.59259259, "global_step/max_steps": "22/60", "elapsed_time": "19m 43s", "remaining_time": "34m 3s", "memory(GiB)": 36.88, "train_speed(s/it)": 53.765253}
|
| 18 |
+
{"loss": 0.02361006, "grad_norm": 1.70428038, "learning_rate": 3.504e-05, "token_acc": 0.99307863, "epoch": 6.0, "global_step/max_steps": "24/60", "elapsed_time": "21m 9s", "remaining_time": "31m 43s", "memory(GiB)": 36.88, "train_speed(s/it)": 52.873533}
|
| 19 |
+
{"eval_loss": 0.09410682, "eval_runtime": 6.2586, "eval_samples_per_second": 0.479, "eval_steps_per_second": 0.479, "eval_token_acc": 0.97533632, "epoch": 6.0, "global_step/max_steps": "24/60", "elapsed_time": "21m 15s", "remaining_time": "31m 53s", "memory(GiB)": 36.88, "train_speed(s/it)": 53.134372}
|
| 20 |
+
{"loss": 0.01686963, "grad_norm": 0.30766147, "learning_rate": 3.246e-05, "token_acc": 0.9943101, "epoch": 6.59259259, "global_step/max_steps": "26/60", "elapsed_time": "23m 17s", "remaining_time": "30m 27s", "memory(GiB)": 36.88, "train_speed(s/it)": 53.734065}
|
| 21 |
+
{"loss": 0.01680034, "grad_norm": 0.65037054, "learning_rate": 2.979e-05, "token_acc": 0.99557889, "epoch": 7.0, "global_step/max_steps": "28/60", "elapsed_time": "24m 43s", "remaining_time": "28m 15s", "memory(GiB)": 36.88, "train_speed(s/it)": 52.973117}
|
| 22 |
+
{"eval_loss": 0.08896798, "eval_runtime": 6.1982, "eval_samples_per_second": 0.484, "eval_steps_per_second": 0.484, "eval_token_acc": 0.97926009, "epoch": 7.0, "global_step/max_steps": "28/60", "elapsed_time": "24m 49s", "remaining_time": "28m 22s", "memory(GiB)": 36.88, "train_speed(s/it)": 53.194533}
|
| 23 |
+
{"loss": 0.00513445, "grad_norm": 0.23172364, "learning_rate": 2.706e-05, "token_acc": 0.99821109, "epoch": 7.59259259, "global_step/max_steps": "30/60", "elapsed_time": "26m 52s", "remaining_time": "26m 52s", "memory(GiB)": 36.88, "train_speed(s/it)": 53.73585}
|
| 24 |
+
{"loss": 0.0113448, "grad_norm": 0.48870251, "learning_rate": 2.431e-05, "token_acc": 0.99687964, "epoch": 8.0, "global_step/max_steps": "32/60", "elapsed_time": "28m 16s", "remaining_time": "24m 44s", "memory(GiB)": 36.88, "train_speed(s/it)": 53.002388}
|
| 25 |
+
{"eval_loss": 0.09867268, "eval_runtime": 6.1729, "eval_samples_per_second": 0.486, "eval_steps_per_second": 0.486, "eval_token_acc": 0.97813901, "epoch": 8.0, "global_step/max_steps": "32/60", "elapsed_time": "28m 22s", "remaining_time": "24m 49s", "memory(GiB)": 36.88, "train_speed(s/it)": 53.195338}
|
| 26 |
+
{"train_runtime": 1706.8733, "train_samples_per_second": 0.237, "train_steps_per_second": 0.035, "total_flos": 7.075103910150298e+16, "train_loss": 0.14959425, "epoch": 8.0, "global_step/max_steps": "32/60", "elapsed_time": "28m 27s", "remaining_time": "24m 54s", "memory(GiB)": 36.88, "train_speed(s/it)": 53.339538}
|
| 27 |
+
{"model_parameter_info": "PeftModelForCausalLM: 15648.3617M Params (466.9112M Trainable [2.9838%]), 0.0001M Buffers.", "last_model_checkpoint": "/workspace/training/checkpoints/panel_counter_v20_27b_sft/v0-20260704-162435/checkpoint-32", "best_model_checkpoint": "/workspace/training/checkpoints/panel_counter_v20_27b_sft/v0-20260704-162435/checkpoint-20", "best_metric": 0.07116517, "global_step": 32, "log_history": [{"loss": 0.6922069787979126, "grad_norm": 1.6430093050003052, "learning_rate": 1.6666666666666667e-05, "token_acc": 0.8433245760687843, "epoch": 0.2962962962962963, "step": 1}, {"loss": 0.5771233439445496, "grad_norm": 1.5347851514816284, "learning_rate": 3.3333333333333335e-05, "token_acc": 0.8648506151142356, "epoch": 0.5925925925925926, "step": 2}, {"loss": 0.5614407658576965, "grad_norm": 1.1821870803833008, "learning_rate": 4.996203791083291e-05, "token_acc": 0.8585475245816802, "epoch": 1.0, "step": 4}, {"eval_loss": 0.41863182187080383, "eval_runtime": 6.1787, "eval_samples_per_second": 0.486, "eval_steps_per_second": 0.486, "eval_token_acc": 0.897982062780269, "epoch": 1.0, "step": 4}, {"loss": 0.3738991916179657, "grad_norm": 1.4124635457992554, "learning_rate": 4.965903258506806e-05, "token_acc": 0.9060283687943262, "epoch": 1.5925925925925926, "step": 6}, {"loss": 0.2040710151195526, "grad_norm": 1.1548813581466675, "learning_rate": 4.905670000773126e-05, "token_acc": 0.9334538284587578, "epoch": 2.0, "step": 8}, {"eval_loss": 0.14844363927841187, "eval_runtime": 6.1727, "eval_samples_per_second": 0.486, "eval_steps_per_second": 0.486, "eval_token_acc": 0.9517937219730942, "epoch": 2.0, "step": 8}, {"loss": 0.13367745280265808, "grad_norm": 0.8823673725128174, "learning_rate": 4.8162351680370044e-05, "token_acc": 0.9616770389780543, "epoch": 2.5925925925925926, "step": 10}, {"loss": 0.12161176651716232, "grad_norm": 0.5558892488479614, "learning_rate": 4.698684378016222e-05, "token_acc": 0.9588641596316193, "epoch": 3.0, "step": 12}, {"eval_loss": 0.09443339705467224, "eval_runtime": 6.19, "eval_samples_per_second": 0.485, "eval_steps_per_second": 0.485, "eval_token_acc": 0.9646860986547086, "epoch": 3.0, "step": 12}, {"loss": 0.07671891897916794, "grad_norm": 0.42686018347740173, "learning_rate": 4.554444538063113e-05, "token_acc": 0.9760853946423073, "epoch": 3.5925925925925926, "step": 14}, {"loss": 0.07431428134441376, "grad_norm": 0.7322053909301758, "learning_rate": 4.385266524442241e-05, "token_acc": 0.9763952116000675, "epoch": 4.0, "step": 16}, {"eval_loss": 0.08191577345132828, "eval_runtime": 6.1494, "eval_samples_per_second": 0.488, "eval_steps_per_second": 0.488, "eval_token_acc": 0.9714125560538116, "epoch": 4.0, "step": 16}, {"loss": 0.0445389561355114, "grad_norm": 0.36494332551956177, "learning_rate": 4.193203929064353e-05, "token_acc": 0.9869766309639727, "epoch": 4.592592592592593, "step": 18}, {"loss": 0.05436675250530243, "grad_norm": 0.5351647734642029, "learning_rate": 3.9805881316624506e-05, "token_acc": 0.9817645481362296, "epoch": 5.0, "step": 20}, {"eval_loss": 0.07116517424583435, "eval_runtime": 6.182, "eval_samples_per_second": 0.485, "eval_steps_per_second": 0.485, "eval_token_acc": 0.9775784753363229, "epoch": 5.0, "step": 20}, {"loss": 0.04044443368911743, "grad_norm": 0.3848143517971039, "learning_rate": 3.7500000000000003e-05, "token_acc": 0.9882840236686391, "epoch": 5.592592592592593, "step": 22}, {"loss": 0.023610057309269905, "grad_norm": 1.7042803764343262, "learning_rate": 3.504238561632424e-05, "token_acc": 0.993078626799557, "epoch": 6.0, "step": 24}, {"eval_loss": 0.09410682320594788, "eval_runtime": 6.2586, "eval_samples_per_second": 0.479, "eval_steps_per_second": 0.479, "eval_token_acc": 0.9753363228699552, "epoch": 6.0, "step": 24}, {"loss": 0.016869625076651573, "grad_norm": 0.3076614737510681, "learning_rate": 3.246287027504237e-05, "token_acc": 0.9943100995732574, "epoch": 6.592592592592593, "step": 26}, {"loss": 0.01680033653974533, "grad_norm": 0.6503705382347107, "learning_rate": 2.9792765798093465e-05, "token_acc": 0.9955788891959104, "epoch": 7.0, "step": 28}, {"eval_loss": 0.08896797895431519, "eval_runtime": 6.1982, "eval_samples_per_second": 0.484, "eval_steps_per_second": 0.484, "eval_token_acc": 0.9792600896860987, "epoch": 7.0, "step": 28}, {"loss": 0.005134452600032091, "grad_norm": 0.2317236363887787, "learning_rate": 2.7064483636808313e-05, "token_acc": 0.998211091234347, "epoch": 7.592592592592593, "step": 30}, {"loss": 0.011344797909259796, "grad_norm": 0.4887025058269501, "learning_rate": 2.4311141440795953e-05, "token_acc": 0.9968796433878158, "epoch": 8.0, "step": 32}, {"eval_loss": 0.09867268055677414, "eval_runtime": 6.1729, "eval_samples_per_second": 0.486, "eval_steps_per_second": 0.486, "eval_token_acc": 0.9781390134529148, "epoch": 8.0, "step": 32}, {"train_runtime": 1706.8733, "train_samples_per_second": 0.237, "train_steps_per_second": 0.035, "total_flos": 7.075103910150298e+16, "train_loss": 0.14959424783592112, "epoch": 8.0, "step": 32}], "memory": 36.880859375}
|