Instructions to use felixwangg/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx3-v2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use felixwangg/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx3-v2 with PEFT:

from peft import PeftModel
from transformers import AutoModelForCausalLM

base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-7B-Instruct")
model = PeftModel.from_pretrained(base_model, "felixwangg/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx3-v2")

Transformers

How to use felixwangg/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx3-v2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="felixwangg/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx3-v2")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("felixwangg/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx3-v2")
model = AutoModelForCausalLM.from_pretrained("felixwangg/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx3-v2", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use felixwangg/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx3-v2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "felixwangg/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx3-v2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "felixwangg/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx3-v2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/felixwangg/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx3-v2

SGLang

How to use felixwangg/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx3-v2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "felixwangg/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx3-v2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "felixwangg/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx3-v2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "felixwangg/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx3-v2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "felixwangg/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx3-v2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use felixwangg/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx3-v2 with Docker Model Runner:
```
docker model run hf.co/felixwangg/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx3-v2
```

Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx3-v2

File size: 156,809 Bytes

0128c85

[2026-04-12 22:28:06,747] [DEBUG] [axolotl.utils.config.log_gpu_memory_usage:127] [PID:1912118] baseline 0.000GB ()
[2026-04-12 22:28:06,748] [INFO] [axolotl.cli.config.load_cfg:259] [PID:1912118] config:
{
  "activation_offloading": false,
  "adapter": "lora",
  "axolotl_config_path": "./axolotl_configs/Qwen2.5-coder-7b-instruct/primevul/lora-primevul-plus-token-diff-mask-ctx3.yaml",
  "base_model": "Qwen/Qwen2.5-Coder-7B-Instruct",
  "base_model_config": "Qwen/Qwen2.5-Coder-7B-Instruct",
  "batch_size": 64,
  "bf16": true,
  "capabilities": {
    "bf16": true,
    "compute_capability": "sm_90",
    "fp8": true,
    "n_gpu": 2,
    "n_node": 1
  },
  "context_parallel_size": 1,
  "dataloader_num_workers": 2,
  "dataloader_pin_memory": true,
  "dataloader_prefetch_factor": 256,
  "dataset_num_proc": 112,
  "dataset_prepared_path": "/home/tkwang/scratch/SecSteer-v2/axolotl-datasets/lora/Qwen2.5-Coder-7B/prime_vul_plus_splitted_token_diff_mask_skip_indent_ctx3_chat_v2",
  "datasets": [
    {
      "message_property_mappings": {
        "content": "content",
        "role": "role"
      },
      "path": "felixwangg/prime_vul_plus_splitted_token_diff_mask_skip_indent_ctx3_chat_v2",
      "split": "train",
      "trust_remote_code": false,
      "type": "pretokenized"
    }
  ],
  "ddp": true,
  "device": "cuda:0",
  "device_map": {
    "": 0
  },
  "dion_rank_fraction": 1.0,
  "dion_rank_multiple_of": 1,
  "early_stopping_patience": 1000,
  "env_capabilities": {
    "torch_version": "2.10.0"
  },
  "eval_batch_size": 4,
  "eval_causal_lm_metrics": [
    "sacrebleu",
    "comet",
    "ter",
    "chrf"
  ],
  "eval_max_new_tokens": 128,
  "eval_sample_packing": false,
  "eval_steps": 15,
  "eval_table_size": 0,
  "experimental_skip_move_to_device": true,
  "flash_attention": true,
  "fp16": false,
  "gradient_accumulation_steps": 8,
  "gradient_checkpointing": true,
  "gradient_checkpointing_kwargs": {
    "use_reentrant": true
  },
  "include_tkps": true,
  "is_falcon_derived_model": false,
  "is_llama_derived_model": false,
  "is_mistral_derived_model": false,
  "learning_rate": 4e-05,
  "lisa_layers_attribute": "model.layers",
  "load_best_model_at_end": true,
  "load_in_4bit": false,
  "load_in_8bit": false,
  "local_rank": 0,
  "logging_steps": 1,
  "lora_alpha": 16,
  "lora_dropout": 0.05,
  "lora_r": 16,
  "lora_target_linear": true,
  "loraplus_lr_embedding": 1e-06,
  "lr_scheduler": "cosine",
  "mean_resizing_embeddings": false,
  "merge_lora": true,
  "micro_batch_size": 4,
  "model_config_type": "qwen2",
  "num_epochs": 1.0,
  "optimizer": "adamw_torch",
  "otel_metrics_host": "localhost",
  "otel_metrics_port": 8000,
  "output_dir": "/home/tkwang/scratch/SecSteer-v2/axolotl-outputs/lora/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx3-v2",
  "pad_to_sequence_len": true,
  "plugins": [
    "diff_mask_trainer.plugin.DiffMaskPlugin"
  ],
  "pretrain_multipack_attn": true,
  "profiler_steps_start": 0,
  "qlora_sharded_model_loading": false,
  "ray_num_workers": 1,
  "resources_per_worker": {
    "GPU": 1
  },
  "sample_packing": false,
  "sample_packing_bin_size": 200,
  "sample_packing_group_size": 100000,
  "save_only_model": false,
  "save_safetensors": true,
  "save_steps": 15,
  "save_total_limit": 1000,
  "sequence_len": 4096,
  "shuffle_before_merging_datasets": false,
  "shuffle_merged_datasets": true,
  "skip_prepare_dataset": false,
  "streaming_multipack_buffer_size": 10000,
  "strict": false,
  "tensor_parallel_size": 1,
  "test_datasets": [
    {
      "message_property_mappings": {
        "content": "content",
        "role": "role"
      },
      "path": "felixwangg/prime_vul_plus_splitted_token_diff_mask_skip_indent_ctx3_chat_v2",
      "split": "validation",
      "trust_remote_code": false,
      "type": "pretokenized"
    }
  ],
  "tf32": false,
  "tiled_mlp_use_original_mlp": true,
  "tokenizer_config": "Qwen/Qwen2.5-Coder-7B-Instruct",
  "tokenizer_save_jinja_files": true,
  "tokenizer_type": "AutoTokenizer",
  "torch_dtype": "torch.bfloat16",
  "train_on_inputs": false,
  "trl": {
    "log_completions": false,
    "mask_truncated_completions": false,
    "ref_model_mixup_alpha": 0.9,
    "ref_model_sync_steps": 64,
    "scale_rewards": true,
    "sync_ref_model": false,
    "use_vllm": false,
    "vllm_server_host": "0.0.0.0",
    "vllm_server_port": 8000
  },
  "type_of_model": "Qwen2ForCausalLM",
  "use_otel_metrics": false,
  "use_ray": false,
  "use_wandb": true,
  "val_set_size": 0.0,
  "vllm": {
    "device": "auto",
    "dtype": "auto",
    "gpu_memory_utilization": 0.9,
    "host": "0.0.0.0",
    "port": 8000
  },
  "wandb_entity": "wtkuan",
  "wandb_log_model": "false",
  "wandb_name": "Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx3-v2",
  "wandb_project": "diff-mask-sft-primevul-ctx-3",
  "wandb_watch": "false",
  "warmup_ratio": 0.1,
  "weight_decay": 0.02,
  "world_size": 2
}
[2026-04-12 22:28:07,450] [DEBUG] [axolotl.loaders.tokenizer.load_tokenizer:285] [PID:1912118] EOS: 151645 / <|im_end|>
[2026-04-12 22:28:07,451] [DEBUG] [axolotl.loaders.tokenizer.load_tokenizer:286] [PID:1912118] BOS: None / None
[2026-04-12 22:28:07,451] [DEBUG] [axolotl.loaders.tokenizer.load_tokenizer:287] [PID:1912118] PAD: 151643 / <|endoftext|>
[2026-04-12 22:28:07,451] [DEBUG] [axolotl.loaders.tokenizer.load_tokenizer:288] [PID:1912118] UNK: None / None
[2026-04-12 22:28:07,564] [INFO] [axolotl.utils.data.shared.load_preprocessed_dataset:475] [PID:1912118] Loading prepared dataset from disk at /home/tkwang/scratch/SecSteer-v2/axolotl-datasets/lora/Qwen2.5-Coder-7B/prime_vul_plus_splitted_token_diff_mask_skip_indent_ctx3_chat_v2/8754561ab695e953202410ad43299050...
[2026-04-12 22:28:07,582] [INFO] [axolotl.utils.data.shared.load_preprocessed_dataset:475] [PID:1912118] Loading prepared dataset from disk at /home/tkwang/scratch/SecSteer-v2/axolotl-datasets/lora/Qwen2.5-Coder-7B/prime_vul_plus_splitted_token_diff_mask_skip_indent_ctx3_chat_v2/7a3c1ae28a530da625e3f3b6a296b5d3...
[2026-04-12 22:28:07,629] [DEBUG] [axolotl.utils.trainer.calculate_total_num_steps:417] [PID:1912118] total_num_tokens: 3_735_150
[2026-04-12 22:28:07,756] [DEBUG] [axolotl.utils.trainer.calculate_total_num_steps:435] [PID:1912118] `total_supervised_tokens: 2_946_329`
[2026-04-12 22:28:07,756] [DEBUG] [axolotl.utils.trainer.calculate_total_num_steps:533] [PID:1912118] total_num_steps: 57
[2026-04-12 22:28:07,756] [INFO] [axolotl.utils.data.sft._prepare_standard_dataset:121] [PID:1912118] Maximum number of steps set at 57
[2026-04-12 22:28:07,775] [DEBUG] [axolotl.train.setup_model_and_tokenizer:70] [PID:1912118] loading tokenizer... Qwen/Qwen2.5-Coder-7B-Instruct
[2026-04-12 22:28:08,328] [DEBUG] [axolotl.loaders.tokenizer.load_tokenizer:285] [PID:1912118] EOS: 151645 / <|im_end|>
[2026-04-12 22:28:08,328] [DEBUG] [axolotl.loaders.tokenizer.load_tokenizer:286] [PID:1912118] BOS: None / None
[2026-04-12 22:28:08,328] [DEBUG] [axolotl.loaders.tokenizer.load_tokenizer:287] [PID:1912118] PAD: 151643 / <|endoftext|>
[2026-04-12 22:28:08,328] [DEBUG] [axolotl.loaders.tokenizer.load_tokenizer:288] [PID:1912118] UNK: None / None
[2026-04-12 22:28:08,328] [DEBUG] [axolotl.train.setup_model_and_tokenizer:82] [PID:1912118] Loading model
[2026-04-12 22:28:08,401] [DEBUG] [axolotl.monkeypatch.transformers.trainer_loss_calc.patch_evaluation_loop:87] [PID:1912118] Patched Trainer.evaluation_loop with nanmean loss calculation
[2026-04-12 22:28:08,402] [DEBUG] [axolotl.monkeypatch.transformers.trainer_loss_calc.patch_maybe_log_save_evaluate:138] [PID:1912118] Patched Trainer._maybe_log_save_evaluate with nanmean loss calculation

Loading checkpoint shards:   0%|                                                                           | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|                                                                           | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|████████████████▊                                                  | 1/4 [00:00<00:02,  1.08it/s]
Loading checkpoint shards:  25%|████████████████▊                                                  | 1/4 [00:00<00:02,  1.08it/s]
Loading checkpoint shards:  50%|█████████████████████████████████▌                                 | 2/4 [00:01<00:01,  1.08it/s]
Loading checkpoint shards:  50%|█████████████████████████████████▌                                 | 2/4 [00:01<00:01,  1.08it/s]
Loading checkpoint shards:  75%|██████████████████████████████████████████████████▎                | 3/4 [00:02<00:00,  1.14it/s]
Loading checkpoint shards:  75%|██████████████████████████████████████████████████▎                | 3/4 [00:02<00:00,  1.14it/s]
Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████| 4/4 [00:02<00:00,  1.67it/s]
Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████| 4/4 [00:02<00:00,  1.41it/s]
Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████| 4/4 [00:02<00:00,  1.67it/s]
Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████| 4/4 [00:02<00:00,  1.41it/s]
[2026-04-12 22:28:12,216] [INFO] [axolotl.loaders.model._configure_embedding_dtypes:347] [PID:1912118] Converting modules to torch.bfloat16
[2026-04-12 22:28:12,217] [DEBUG] [axolotl.loaders.model.log_gpu_memory_usage:127] [PID:1912118] Memory usage after model load 17.233GB (+17.233GB allocated, +18.252GB reserved)
[2026-04-12 22:28:12,218] [INFO] [axolotl.loaders.adapter.load_lora:81] [PID:1912118] found linear modules: ['down_proj', 'gate_proj', 'k_proj', 'o_proj', 'q_proj', 'up_proj', 'v_proj']
trainable params: 40,370,176 || all params: 7,655,986,688 || trainable%: 0.5273
[2026-04-12 22:28:12,492] [DEBUG] [axolotl.loaders.model.log_gpu_memory_usage:127] [PID:1912118] after adapters 14.337GB (+14.337GB allocated, +18.330GB reserved)
[2026-04-12 22:28:16,196] [WARNING] [py.warnings._showwarnmsg:112] [PID:1912118] /scratch/tkwang/SecSteer-v2/.venv/lib/python3.12/site-packages/trl/extras/vllm_client.py:37: UserWarning: TRL currently supports vLLM versions: 0.10.2, 0.11.0, 0.11.1, 0.11.2, 0.12.0. You have version 0.19.0 installed. We recommend installing a supported version to avoid compatibility issues.
  if is_vllm_available():


[2026-04-12 22:28:16,197] [WARNING] [py.warnings._showwarnmsg:112] [PID:1912119] /scratch/tkwang/SecSteer-v2/.venv/lib/python3.12/site-packages/trl/extras/vllm_client.py:37: UserWarning: TRL currently supports vLLM versions: 0.10.2, 0.11.0, 0.11.1, 0.11.2, 0.12.0. You have version 0.19.0 installed. We recommend installing a supported version to avoid compatibility issues.
  if is_vllm_available():

[2026-04-12 22:28:16,770] [WARNING] [py.warnings._showwarnmsg:112] [PID:1912118] /scratch/tkwang/SecSteer-v2/.venv/lib/python3.12/site-packages/trl/trainer/grpo_trainer.py:105: UserWarning: TRL currently supports vLLM versions: 0.10.2, 0.11.0, 0.11.1, 0.11.2, 0.12.0. You have version 0.19.0 installed. We recommend installing a supported version to avoid compatibility issues.
  if is_vllm_available():

[2026-04-12 22:28:16,772] [WARNING] [py.warnings._showwarnmsg:112] [PID:1912119] /scratch/tkwang/SecSteer-v2/.venv/lib/python3.12/site-packages/trl/trainer/grpo_trainer.py:105: UserWarning: TRL currently supports vLLM versions: 0.10.2, 0.11.0, 0.11.1, 0.11.2, 0.12.0. You have version 0.19.0 installed. We recommend installing a supported version to avoid compatibility issues.
  if is_vllm_available():

DiffMaskPlugin: patching trainer with alpha=1.0
DiffMaskPlugin: compute_loss and prediction_step patched
[2026-04-12 22:28:26,661] [INFO] [axolotl.train.save_initial_configs:413] [PID:1912118] Pre-saving adapter config to /home/tkwang/scratch/SecSteer-v2/axolotl-outputs/lora/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx3-v2...
[2026-04-12 22:28:26,681] [INFO] [axolotl.train.save_initial_configs:417] [PID:1912118] Pre-saving tokenizer to /home/tkwang/scratch/SecSteer-v2/axolotl-outputs/lora/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx3-v2...
[2026-04-12 22:28:26,908] [INFO] [axolotl.train.save_initial_configs:422] [PID:1912118] Pre-saving model config to /home/tkwang/scratch/SecSteer-v2/axolotl-outputs/lora/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx3-v2...
[2026-04-12 22:28:26,927] [INFO] [axolotl.train.execute_training:212] [PID:1912118] Starting trainer...
DiffMaskPlugin: patching trainer with alpha=1.0
DiffMaskPlugin: compute_loss and prediction_step patched
[34m[1mwandb[0m: [wandb.login()] Loaded credentials for https://api.wandb.ai from /home/tkwang/.netrc.
[34m[1mwandb[0m: Currently logged in as: [33mwtkuan[0m to [32mhttps://api.wandb.ai[0m. Use [1m`wandb login --relogin`[0m to force relogin
[34m[1mwandb[0m: [38;5;178m⢿[0m Waiting for wandb.init()...

[Am[2K
[34m[1mwandb[0m: [38;5;178m⣻[0m Waiting for wandb.init()...

[Am[2K
[34m[1mwandb[0m: Tracking run with wandb version 0.24.0
[34m[1mwandb[0m: Run data is saved locally in [35m[1m/scratch/tkwang/SecSteer-v2/wandb/run-20260412_222828-jhtegn10[0m
[34m[1mwandb[0m: Run [1m`wandb offline`[0m to turn off syncing.
[34m[1mwandb[0m: Syncing run [33mQwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx3-v2[0m
[34m[1mwandb[0m: ⭐️ View project at [34m[4mhttps://wandb.ai/wtkuan/diff-mask-sft-primevul-ctx-3[0m
[34m[1mwandb[0m: 🚀 View run at [34m[4mhttps://wandb.ai/wtkuan/diff-mask-sft-primevul-ctx-3/runs/jhtegn10[0m
[34m[1mwandb[0m: Detected [huggingface_hub.inference, mcp, openai] in use.
[34m[1mwandb[0m: Use W&B Weave for improved LLM call tracing. Install Weave with `pip install weave` then add `import weave` to the top of your script.
[34m[1mwandb[0m: For more information, check out the docs at: https://weave-docs.wandb.ai/
[34m[1mwandb[0m: [33mWARNING[0m Saving files without folders. If you want to preserve subdirectories pass base_path to wandb.save, i.e. wandb.save("/mnt/folder/file.h5", base_path="/mnt")
[34m[1mwandb[0m: [33mWARNING[0m Symlinked 1 file into the W&B run directory; call wandb.save again to sync new files.
[2026-04-12 22:28:31,422] [INFO] [axolotl.utils.callbacks.on_train_begin:757] [PID:1912118] The Axolotl config has been saved to the WandB run under files.

  0%|                                                                                                     | 0/57 [00:00<?, ?it/s][2026-04-12 22:28:31,426] [INFO] [axolotl.core.trainers.base.evaluate:400] [PID:1912118] Running evaluation step...


  0%|                                                                                                     | 0/50 [00:00<?, ?it/s][A

  4%|███▋                                                                                         | 2/50 [00:00<00:13,  3.47it/s][A

  6%|█████▌                                                                                       | 3/50 [00:01<00:20,  2.31it/s][A

  8%|███████▍                                                                                     | 4/50 [00:01<00:23,  1.93it/s][A

 10%|█████████▎                                                                                   | 5/50 [00:02<00:25,  1.79it/s][A

 12%|███████████▏                                                                                 | 6/50 [00:03<00:25,  1.71it/s][A

 14%|█████████████                                                                                | 7/50 [00:03<00:25,  1.67it/s][A

 16%|██████████████▉                                                                              | 8/50 [00:04<00:25,  1.64it/s][A

 18%|████████████████▋                                                                            | 9/50 [00:05<00:25,  1.59it/s][A

 20%|██████████████████▍                                                                         | 10/50 [00:05<00:25,  1.59it/s][A

 22%|████████████████████▏                                                                       | 11/50 [00:06<00:24,  1.59it/s][A

 24%|██████████████████████                                                                      | 12/50 [00:06<00:24,  1.57it/s][A

 26%|███████████████████████▉                                                                    | 13/50 [00:07<00:23,  1.58it/s][A

 28%|█████████████████████████▊                                                                  | 14/50 [00:08<00:22,  1.58it/s][A

 30%|███████████████████████████▌                                                                | 15/50 [00:08<00:22,  1.58it/s][A

 32%|█████████████████████████████▍                                                              | 16/50 [00:09<00:21,  1.58it/s][A

 34%|███████████████████████████████▎                                                            | 17/50 [00:10<00:21,  1.55it/s][A

 36%|█████████████████████████████████                                                           | 18/50 [00:10<00:20,  1.56it/s][A

 38%|██████████████████████████████████▉                                                         | 19/50 [00:11<00:19,  1.56it/s][A

 40%|████████████████████████████████████▊                                                       | 20/50 [00:12<00:19,  1.57it/s][A

 42%|██████████████████████████████████████▋                                                     | 21/50 [00:12<00:18,  1.57it/s][A

 44%|████████████████████████████████████████▍                                                   | 22/50 [00:13<00:17,  1.57it/s][A

 46%|██████████████████████████████████████████▎                                                 | 23/50 [00:13<00:17,  1.57it/s][A

 48%|████████████████████████████████████████████▏                                               | 24/50 [00:14<00:16,  1.58it/s][A

 50%|██████████████████████████████████████████████                                              | 25/50 [00:15<00:16,  1.55it/s][A

 52%|███████████████████████████████████████████████▊                                            | 26/50 [00:15<00:15,  1.56it/s][A

 54%|█████████████████████████████████████████████████▋                                          | 27/50 [00:16<00:14,  1.57it/s][A

 56%|███████████████████████████████████████████████████▌                                        | 28/50 [00:17<00:14,  1.57it/s][A

 58%|█████████████████████████████████████████████████████▎                                      | 29/50 [00:17<00:13,  1.57it/s][A

 60%|███████████████████████████████████████████████████████▏                                    | 30/50 [00:18<00:12,  1.57it/s][A

 62%|█████████████████████████████████████████████████████████                                   | 31/50 [00:19<00:12,  1.57it/s][A

 64%|██████████████████████████████████████████████████████████▉                                 | 32/50 [00:19<00:11,  1.58it/s][A

 66%|████████████████████████████████████████████████████████████▋                               | 33/50 [00:20<00:10,  1.55it/s][A

 68%|██████████████████████████████████████████████████████████████▌                             | 34/50 [00:21<00:10,  1.56it/s][A

 70%|████████████████████████████████████████████████████████████████▍                           | 35/50 [00:21<00:09,  1.56it/s][A

 72%|██████████████████████████████████████████████████████████████████▏                         | 36/50 [00:22<00:08,  1.57it/s][A

 74%|████████████████████████████████████████████████████████████████████                        | 37/50 [00:22<00:08,  1.57it/s][A

 76%|█████████████████████████████████████████████████████████████████████▉                      | 38/50 [00:23<00:07,  1.58it/s][A

 78%|███████████████████████████████████████████████████████████████████████▊                    | 39/50 [00:24<00:06,  1.57it/s][A

 80%|█████████████████████████████████████████████████████████████████████████▌                  | 40/50 [00:24<00:06,  1.58it/s][A

 82%|███████████████████████████████████████████████████████████████████████████▍                | 41/50 [00:25<00:05,  1.53it/s][A

 84%|█████████████████████████████████████████████████████████████████████████████▎              | 42/50 [00:26<00:05,  1.55it/s][A

 86%|███████████████████████████████████████████████████████████████████████████████             | 43/50 [00:26<00:04,  1.55it/s][A

 88%|████████████████████████████████████████████████████████████████████████████████▉           | 44/50 [00:27<00:03,  1.56it/s][A

 90%|██████████████████████████████████████████████████████████████████████████████████▊         | 45/50 [00:28<00:03,  1.56it/s][A

 92%|████████████████████████████████████████████████████████████████████████████████████▋       | 46/50 [00:28<00:02,  1.57it/s][A

 94%|██████████████████████████████████████████████████████████████████████████████████████▍     | 47/50 [00:29<00:01,  1.57it/s][A

 96%|████████████████████████████████████████████████████████████████████████████████████████▎   | 48/50 [00:29<00:01,  1.57it/s][A

 98%|██████████████████████████████████████████████████████████████████████████████████████████▏ | 49/50 [00:30<00:00,  1.55it/s][ATraceback (most recent call last):
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
    finalizer()
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
    res = self._callback(*self._args, **self._kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
    rmtree(tempdir, onerror=onerror)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
    _rmtree_safe_fd(stack, onexc)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
    onexc(func, path, err)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
    return onerror(func, path, exc_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
    onexc(os.unlink, fullname, err)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
    return onerror(func, path, exc_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
    os.unlink(entry.name, dir_fd=topfd)
OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-qo8f6k6o'
Traceback (most recent call last):
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
    finalizer()
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
    res = self._callback(*self._args, **self._kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
    rmtree(tempdir, onerror=onerror)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
    _rmtree_safe_fd(stack, onexc)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
    onexc(func, path, err)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
    return onerror(func, path, exc_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
    onexc(os.unlink, fullname, err)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
    return onerror(func, path, exc_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
    os.unlink(entry.name, dir_fd=topfd)
OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-bmoa4jkb'
Traceback (most recent call last):
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
    finalizer()
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
    res = self._callback(*self._args, **self._kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
    rmtree(tempdir, onerror=onerror)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
    _rmtree_safe_fd(stack, onexc)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
    onexc(func, path, err)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
    return onerror(func, path, exc_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
    onexc(os.unlink, fullname, err)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
    return onerror(func, path, exc_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
    os.unlink(entry.name, dir_fd=topfd)
OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-rxze8741'
Traceback (most recent call last):
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
    finalizer()
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
    res = self._callback(*self._args, **self._kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
    rmtree(tempdir, onerror=onerror)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
    _rmtree_safe_fd(stack, onexc)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
    onexc(func, path, err)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
    return onerror(func, path, exc_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
    onexc(os.unlink, fullname, err)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
    return onerror(func, path, exc_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
    os.unlink(entry.name, dir_fd=topfd)
OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-bvnotu57'


100%|████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:31<00:00,  1.51it/s][A
                                                                                                                                 

                                                                                                                                 
[A{'eval_loss': 0.8633460402488708, 'eval_runtime': 32.5395, 'eval_samples_per_second': 12.293, 'eval_steps_per_second': 1.537, 'eval_ppl': 2.37108, 'memory/max_active (GiB)': 42.36, 'memory/max_allocated (GiB)': 42.36, 'memory/device_reserved (GiB)': 52.88, 'epoch': 0}

  0%|                                                                                                     | 0/57 [00:32<?, ?it/s]

100%|████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:31<00:00,  1.51it/s][A

                                                                                                                                 [A
  2%|█▋                                                                                           | 1/57 [00:52<48:37, 52.09s/it]
                                                                                                                                 
{'loss': 7.6116, 'grad_norm': 0.5041501522064209, 'learning_rate': 0.0, 'ppl': 2021.50993, 'memory/max_active (GiB)': 45.83, 'memory/max_allocated (GiB)': 45.83, 'memory/device_reserved (GiB)': 60.59, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.02}

  2%|█▋                                                                                           | 1/57 [00:52<48:37, 52.09s/it]
  4%|███▎                                                                                         | 2/57 [01:09<28:57, 31.60s/it]
                                                                                                                                 
{'loss': 6.4408, 'grad_norm': 0.4618247151374817, 'learning_rate': 8.000000000000001e-06, 'ppl': 626.90813, 'memory/max_active (GiB)': 46.13, 'memory/max_allocated (GiB)': 46.13, 'memory/device_reserved (GiB)': 60.61, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.04}

  4%|███▎                                                                                         | 2/57 [01:09<28:57, 31.60s/it]
  5%|████▉                                                                                        | 3/57 [01:26<22:41, 25.21s/it]
                                                                                                                                 
{'loss': 7.0865, 'grad_norm': 0.44204896688461304, 'learning_rate': 1.6000000000000003e-05, 'ppl': 1195.71546, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 60.61, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.05}

  5%|████▉                                                                                        | 3/57 [01:26<22:41, 25.21s/it]
  7%|██████▌                                                                                      | 4/57 [01:44<19:31, 22.11s/it]
                                                                                                                                 
{'loss': 7.5738, 'grad_norm': 0.5431805849075317, 'learning_rate': 2.4e-05, 'ppl': 1946.52303, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 60.61, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.07}

  7%|██████▌                                                                                      | 4/57 [01:44<19:31, 22.11s/it]
  9%|████████▏                                                                                    | 5/57 [02:01<17:42, 20.42s/it]
                                                                                                                                 
{'loss': 7.103, 'grad_norm': 0.4767451286315918, 'learning_rate': 3.2000000000000005e-05, 'ppl': 1215.60844, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 60.61, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.09}

  9%|████████▏                                                                                    | 5/57 [02:01<17:42, 20.42s/it]
 11%|█████████▊                                                                                   | 6/57 [02:19<16:31, 19.43s/it]
                                                                                                                                 
{'loss': 6.562, 'grad_norm': 0.5277318358421326, 'learning_rate': 4e-05, 'ppl': 707.68565, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 60.61, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.11}

 11%|█████████▊                                                                                   | 6/57 [02:19<16:31, 19.43s/it]
 12%|███████████▍                                                                                 | 7/57 [02:36<15:36, 18.74s/it]
                                                                                                                                 
{'loss': 6.8916, 'grad_norm': 0.5900119543075562, 'learning_rate': 3.996351108446635e-05, 'ppl': 983.97452, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 60.61, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.12}

 12%|███████████▍                                                                                 | 7/57 [02:36<15:36, 18.74s/it]
 14%|█████████████                                                                                | 8/57 [02:54<14:57, 18.32s/it]
                                                                                                                                 
{'loss': 7.2573, 'grad_norm': 0.5291950106620789, 'learning_rate': 3.985417748196108e-05, 'ppl': 1418.42162, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 60.61, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.14}

 14%|█████████████                                                                                | 8/57 [02:54<14:57, 18.32s/it]
 16%|██████████████▋                                                                              | 9/57 [03:11<14:27, 18.07s/it]
                                                                                                                                 
{'loss': 6.6687, 'grad_norm': 0.5454150438308716, 'learning_rate': 3.967239813894288e-05, 'ppl': 787.37136, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 60.61, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.16}

 16%|██████████████▋                                                                              | 9/57 [03:11<14:27, 18.07s/it]
 18%|████████████████▏                                                                           | 10/57 [03:28<13:58, 17.84s/it]
                                                                                                                                 
{'loss': 7.162, 'grad_norm': 0.6699212789535522, 'learning_rate': 3.9418836348521045e-05, 'ppl': 1289.48733, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 60.61, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.18}

 18%|████████████████▏                                                                           | 10/57 [03:28<13:58, 17.84s/it]
 19%|█████████████████▊                                                                          | 11/57 [03:46<13:33, 17.69s/it]
                                                                                                                                 
{'loss': 6.4958, 'grad_norm': 0.6699477434158325, 'learning_rate': 3.909441733017092e-05, 'ppl': 662.3539, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 60.61, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.19}

 19%|█████████████████▊                                                                          | 11/57 [03:46<13:33, 17.69s/it]
 21%|███████████████████▎                                                                        | 12/57 [04:03<13:12, 17.62s/it]
                                                                                                                                 
{'loss': 6.8094, 'grad_norm': 0.6991468667984009, 'learning_rate': 3.8700324853708304e-05, 'ppl': 906.32685, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 60.61, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.21}

 21%|███████████████████▎                                                                        | 12/57 [04:03<13:12, 17.62s/it]
 23%|████████████████████▉                                                                       | 13/57 [04:21<12:52, 17.56s/it]
                                                                                                                                 
{'loss': 7.1837, 'grad_norm': 0.7107539772987366, 'learning_rate': 3.82379969198418e-05, 'ppl': 1317.77502, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 60.61, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.23}

 23%|████████████████████▉                                                                       | 13/57 [04:21<12:52, 17.56s/it]
 25%|██████████████████████▌                                                                     | 14/57 [04:38<12:33, 17.51s/it]
                                                                                                                                 
{'loss': 6.6232, 'grad_norm': 0.6694767475128174, 'learning_rate': 3.7709120513064196e-05, 'ppl': 752.34877, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 60.61, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.25}

 25%|██████████████████████▌                                                                     | 14/57 [04:38<12:33, 17.51s/it]
 26%|████████████████████████▏                                                                   | 15/57 [04:55<12:14, 17.49s/it]
                                                                                                                                 
{'loss': 6.3713, 'grad_norm': 0.679179847240448, 'learning_rate': 3.711562544602895e-05, 'ppl': 584.8176, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 60.61, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.26}

 26%|████████████████████████▏                                                                   | 15/57 [04:55<12:14, 17.49s/it][2026-04-12 22:33:27,332] [INFO] [axolotl.core.trainers.base.evaluate:400] [PID:1912118] Running evaluation step...


  0%|                                                                                                     | 0/50 [00:00<?, ?it/s][A

  4%|███▋                                                                                         | 2/50 [00:00<00:15,  3.12it/s][A

  6%|█████▌                                                                                       | 3/50 [00:01<00:21,  2.21it/s][A

  8%|███████▍                                                                                     | 4/50 [00:01<00:24,  1.89it/s][A

 10%|█████████▎                                                                                   | 5/50 [00:02<00:25,  1.77it/s][A

 12%|███████████▏                                                                                 | 6/50 [00:03<00:25,  1.70it/s][A

 14%|█████████████                                                                                | 7/50 [00:03<00:25,  1.66it/s][A

 16%|██████████████▉                                                                              | 8/50 [00:04<00:25,  1.63it/s][A

 18%|████████████████▋                                                                            | 9/50 [00:05<00:26,  1.58it/s][A

 20%|██████████████████▍                                                                         | 10/50 [00:05<00:25,  1.58it/s][A

 22%|████████████████████▏                                                                       | 11/50 [00:06<00:24,  1.58it/s][A

 24%|██████████████████████                                                                      | 12/50 [00:07<00:24,  1.57it/s][A

 26%|███████████████████████▉                                                                    | 13/50 [00:07<00:23,  1.57it/s][A

 28%|█████████████████████████▊                                                                  | 14/50 [00:08<00:22,  1.57it/s][A

 30%|███████████████████████████▌                                                                | 15/50 [00:08<00:22,  1.58it/s][A

 32%|█████████████████████████████▍                                                              | 16/50 [00:09<00:21,  1.58it/s][A

 34%|███████████████████████████████▎                                                            | 17/50 [00:10<00:21,  1.55it/s][A

 36%|█████████████████████████████████                                                           | 18/50 [00:10<00:20,  1.56it/s][A

 38%|██████████████████████████████████▉                                                         | 19/50 [00:11<00:19,  1.56it/s][A

 40%|████████████████████████████████████▊                                                       | 20/50 [00:12<00:19,  1.56it/s][A

 42%|██████████████████████████████████████▋                                                     | 21/50 [00:12<00:18,  1.56it/s][A

 44%|████████████████████████████████████████▍                                                   | 22/50 [00:13<00:17,  1.57it/s][A

 46%|██████████████████████████████████████████▎                                                 | 23/50 [00:14<00:17,  1.57it/s][A

 48%|████████████████████████████████████████████▏                                               | 24/50 [00:14<00:16,  1.57it/s][A

 50%|██████████████████████████████████████████████                                              | 25/50 [00:15<00:16,  1.55it/s][A

 52%|███████████████████████████████████████████████▊                                            | 26/50 [00:16<00:15,  1.56it/s][A

 54%|█████████████████████████████████████████████████▋                                          | 27/50 [00:16<00:14,  1.56it/s][A

 56%|███████████████████████████████████████████████████▌                                        | 28/50 [00:17<00:14,  1.57it/s][A

 58%|█████████████████████████████████████████████████████▎                                      | 29/50 [00:17<00:13,  1.56it/s][A

 60%|███████████████████████████████████████████████████████▏                                    | 30/50 [00:18<00:12,  1.57it/s][A

 62%|█████████████████████████████████████████████████████████                                   | 31/50 [00:19<00:12,  1.57it/s][A

 64%|██████████████████████████████████████████████████████████▉                                 | 32/50 [00:19<00:11,  1.58it/s][A

 66%|████████████████████████████████████████████████████████████▋                               | 33/50 [00:20<00:10,  1.55it/s][A

 68%|██████████████████████████████████████████████████████████████▌                             | 34/50 [00:21<00:10,  1.56it/s][A

 70%|████████████████████████████████████████████████████████████████▍                           | 35/50 [00:21<00:09,  1.56it/s][A

 72%|██████████████████████████████████████████████████████████████████▏                         | 36/50 [00:22<00:08,  1.57it/s][A

 74%|████████████████████████████████████████████████████████████████████                        | 37/50 [00:23<00:08,  1.57it/s][A

 76%|█████████████████████████████████████████████████████████████████████▉                      | 38/50 [00:23<00:07,  1.57it/s][A

 78%|███████████████████████████████████████████████████████████████████████▊                    | 39/50 [00:24<00:07,  1.57it/s][A

 80%|█████████████████████████████████████████████████████████████████████████▌                  | 40/50 [00:24<00:06,  1.57it/s][A

 82%|███████████████████████████████████████████████████████████████████████████▍                | 41/50 [00:25<00:05,  1.53it/s][A

 84%|█████████████████████████████████████████████████████████████████████████████▎              | 42/50 [00:26<00:05,  1.55it/s][A

 86%|███████████████████████████████████████████████████████████████████████████████             | 43/50 [00:26<00:04,  1.55it/s][A

 88%|████████████████████████████████████████████████████████████████████████████████▉           | 44/50 [00:27<00:03,  1.56it/s][A

 90%|██████████████████████████████████████████████████████████████████████████████████▊         | 45/50 [00:28<00:03,  1.56it/s][A

 92%|████████████████████████████████████████████████████████████████████████████████████▋       | 46/50 [00:28<00:02,  1.56it/s][A

 94%|██████████████████████████████████████████████████████████████████████████████████████▍     | 47/50 [00:29<00:01,  1.57it/s][A

 96%|████████████████████████████████████████████████████████████████████████████████████████▎   | 48/50 [00:30<00:01,  1.57it/s][A

 98%|██████████████████████████████████████████████████████████████████████████████████████████▏ | 49/50 [00:30<00:00,  1.55it/s][ATraceback (most recent call last):
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
    finalizer()
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
    res = self._callback(*self._args, **self._kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
    rmtree(tempdir, onerror=onerror)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
    _rmtree_safe_fd(stack, onexc)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
    onexc(func, path, err)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
    return onerror(func, path, exc_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
    onexc(os.unlink, fullname, err)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
    return onerror(func, path, exc_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
    os.unlink(entry.name, dir_fd=topfd)
OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-3ucrsz0p'
Traceback (most recent call last):
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
    finalizer()
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
    res = self._callback(*self._args, **self._kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
    rmtree(tempdir, onerror=onerror)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
    _rmtree_safe_fd(stack, onexc)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
    onexc(func, path, err)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
    return onerror(func, path, exc_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
    onexc(os.unlink, fullname, err)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
    return onerror(func, path, exc_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
    os.unlink(entry.name, dir_fd=topfd)
OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-4ugcbvwj'
Traceback (most recent call last):
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
    finalizer()
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
    res = self._callback(*self._args, **self._kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
    rmtree(tempdir, onerror=onerror)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
    _rmtree_safe_fd(stack, onexc)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
    onexc(func, path, err)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
    return onerror(func, path, exc_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
    onexc(os.unlink, fullname, err)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
    return onerror(func, path, exc_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
    os.unlink(entry.name, dir_fd=topfd)
OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-pcg7qm29'
Traceback (most recent call last):
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
    finalizer()
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
    res = self._callback(*self._args, **self._kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
    rmtree(tempdir, onerror=onerror)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
    _rmtree_safe_fd(stack, onexc)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
    onexc(func, path, err)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
    return onerror(func, path, exc_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
    onexc(os.unlink, fullname, err)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
    return onerror(func, path, exc_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
    os.unlink(entry.name, dir_fd=topfd)
OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-bzt7vbep'


100%|████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:31<00:00,  1.52it/s][A
                                                                                                                                 

                                                                                                                                 
[A{'eval_loss': 0.8177846670150757, 'eval_runtime': 32.4169, 'eval_samples_per_second': 12.339, 'eval_steps_per_second': 1.542, 'eval_ppl': 2.26548, 'memory/max_active (GiB)': 42.7, 'memory/max_allocated (GiB)': 42.7, 'memory/device_reserved (GiB)': 60.61, 'epoch': 0.26, 'tokens/train_per_sec_per_gpu': 0.0}

 26%|████████████████████████▏                                                                   | 15/57 [05:28<12:14, 17.49s/it]

100%|████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:31<00:00,  1.52it/s][A

                                                                                                                                 [A[2026-04-12 22:33:59,980] [INFO] [axolotl.core.trainers.base._save:721] [PID:1912118] Saving model checkpoint to /home/tkwang/scratch/SecSteer-v2/axolotl-outputs/lora/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx3-v2/checkpoint-15
[2026-04-12 22:34:00,954] [WARNING] [py.warnings._showwarnmsg:112] [PID:1912118] /scratch/tkwang/SecSteer-v2/.venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
  return func(*args, **kwargs)


 28%|█████████████████████████▊                                                                  | 16/57 [05:47<19:01, 27.85s/it]
                                                                                                                                 
{'loss': 6.788, 'grad_norm': 0.7090097665786743, 'learning_rate': 3.645967731787313e-05, 'ppl': 887.13751, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.76, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.28}

 28%|█████████████████████████▊                                                                  | 16/57 [05:47<19:01, 27.85s/it]
 30%|███████████████████████████▍                                                                | 17/57 [06:05<16:28, 24.70s/it]
                                                                                                                                 
{'loss': 6.1358, 'grad_norm': 0.6638818383216858, 'learning_rate': 3.5743669612181004e-05, 'ppl': 462.10863, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.76, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.3}

 30%|███████████████████████████▍                                                                | 17/57 [06:05<16:28, 24.70s/it]
 32%|█████████████████████████████                                                               | 18/57 [06:22<14:36, 22.48s/it]
                                                                                                                                 
{'loss': 6.4475, 'grad_norm': 0.6962721943855286, 'learning_rate': 3.497021496342203e-05, 'ppl': 631.12251, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.76, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.32}

 32%|█████████████████████████████                                                               | 18/57 [06:22<14:36, 22.48s/it]
 33%|██████████████████████████████▋                                                             | 19/57 [06:39<13:17, 20.98s/it]
                                                                                                                                 
{'loss': 7.3256, 'grad_norm': 0.5690726041793823, 'learning_rate': 3.4142135623730954e-05, 'ppl': 1518.68484, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.76, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.33}

 33%|██████████████████████████████▋                                                             | 19/57 [06:39<13:17, 20.98s/it]
 35%|████████████████████████████████▎                                                           | 20/57 [06:57<12:16, 19.91s/it]
                                                                                                                                 
{'loss': 6.4585, 'grad_norm': 0.554060697555542, 'learning_rate': 3.326245316481591e-05, 'ppl': 638.10318, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.76, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.35}

 35%|████████████████████████████████▎                                                           | 20/57 [06:57<12:16, 19.91s/it]
 37%|█████████████████████████████████▉                                                          | 21/57 [07:14<11:28, 19.14s/it]
                                                                                                                                 
{'loss': 6.109, 'grad_norm': 0.5055751800537109, 'learning_rate': 3.2334377452570866e-05, 'ppl': 449.8886, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.76, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.37}

 37%|█████████████████████████████████▉                                                          | 21/57 [07:14<11:28, 19.14s/it]
 39%|███████████████████████████████████▌                                                        | 22/57 [07:32<10:51, 18.62s/it]
                                                                                                                                 
{'loss': 6.1619, 'grad_norm': 0.48670968413352966, 'learning_rate': 3.136129493462312e-05, 'ppl': 474.32844, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.76, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.39}

 39%|███████████████████████████████████▌                                                        | 22/57 [07:32<10:51, 18.62s/it]
 40%|█████████████████████████████████████                                                       | 23/57 [07:49<10:20, 18.26s/it]
                                                                                                                                 
{'loss': 6.3872, 'grad_norm': 0.4474429488182068, 'learning_rate': 3.0346756283553138e-05, 'ppl': 594.19051, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.76, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.4}

 40%|█████████████████████████████████████                                                       | 23/57 [07:49<10:20, 18.26s/it]
 42%|██████████████████████████████████████▋                                                     | 24/57 [08:07<09:54, 18.02s/it]
                                                                                                                                 
{'loss': 5.978, 'grad_norm': 0.5226128697395325, 'learning_rate': 2.9294463440875375e-05, 'ppl': 394.65028, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.76, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.42}

 42%|██████████████████████████████████████▋                                                     | 24/57 [08:07<09:54, 18.02s/it]
 44%|████████████████████████████████████████▎                                                   | 25/57 [08:24<09:30, 17.84s/it]
                                                                                                                                 
{'loss': 5.8658, 'grad_norm': 0.4168238639831543, 'learning_rate': 2.820825610905514e-05, 'ppl': 352.76425, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.76, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.44}

 44%|████████████████████████████████████████▎                                                   | 25/57 [08:24<09:30, 17.84s/it]
 46%|█████████████████████████████████████████▉                                                  | 26/57 [08:41<09:08, 17.69s/it]
                                                                                                                                 
{'loss': 6.4017, 'grad_norm': 0.4784623682498932, 'learning_rate': 2.7092097740850712e-05, 'ppl': 602.86904, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.76, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.46}

 46%|█████████████████████████████████████████▉                                                  | 26/57 [08:41<09:08, 17.69s/it]
 47%|███████████████████████████████████████████▌                                                | 27/57 [08:59<08:50, 17.67s/it]
                                                                                                                                 
{'loss': 6.2963, 'grad_norm': 0.4028608500957489, 'learning_rate': 2.595006107710406e-05, 'ppl': 542.56072, 'memory/max_active (GiB)': 46.13, 'memory/max_allocated (GiB)': 46.13, 'memory/device_reserved (GiB)': 61.76, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.47}

 47%|███████████████████████████████████████████▌                                                | 27/57 [08:59<08:50, 17.67s/it]
 49%|█████████████████████████████████████████████▏                                              | 28/57 [09:16<08:31, 17.63s/it]
                                                                                                                                 
{'loss': 6.181, 'grad_norm': 0.40923628211021423, 'learning_rate': 2.4786313285751158e-05, 'ppl': 483.47519, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.76, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.49}

 49%|█████████████████████████████████████████████▏                                              | 28/57 [09:16<08:31, 17.63s/it]
 51%|██████████████████████████████████████████████▊                                             | 29/57 [09:34<08:10, 17.53s/it]
                                                                                                                                 
{'loss': 6.081, 'grad_norm': 0.4763945937156677, 'learning_rate': 2.360510075627812e-05, 'ppl': 437.46644, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.76, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.51}

 51%|██████████████████████████████████████████████▊                                             | 29/57 [09:34<08:10, 17.53s/it]
 53%|████████████████████████████████████████████████▍                                           | 30/57 [09:51<07:52, 17.51s/it]
                                                                                                                                 
{'loss': 6.3417, 'grad_norm': 0.44060218334198, 'learning_rate': 2.2410733605106462e-05, 'ppl': 567.76068, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.76, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.53}

 53%|████████████████████████████████████████████████▍                                           | 30/57 [09:51<07:52, 17.51s/it][2026-04-12 22:38:23,111] [INFO] [axolotl.core.trainers.base.evaluate:400] [PID:1912118] Running evaluation step...


  0%|                                                                                                     | 0/50 [00:00<?, ?it/s][A

  4%|███▋                                                                                         | 2/50 [00:00<00:15,  3.12it/s][A

  6%|█████▌                                                                                       | 3/50 [00:01<00:21,  2.22it/s][A

  8%|███████▍                                                                                     | 4/50 [00:01<00:24,  1.88it/s][A

 10%|█████████▎                                                                                   | 5/50 [00:02<00:25,  1.77it/s][A

 12%|███████████▏                                                                                 | 6/50 [00:03<00:25,  1.70it/s][A

 14%|█████████████                                                                                | 7/50 [00:03<00:25,  1.66it/s][A

 16%|██████████████▉                                                                              | 8/50 [00:04<00:25,  1.63it/s][A

 18%|████████████████▋                                                                            | 9/50 [00:05<00:26,  1.57it/s][A

 20%|██████████████████▍                                                                         | 10/50 [00:05<00:25,  1.58it/s][A

 22%|████████████████████▏                                                                       | 11/50 [00:06<00:24,  1.58it/s][A

 24%|██████████████████████                                                                      | 12/50 [00:07<00:24,  1.57it/s][A

 26%|███████████████████████▉                                                                    | 13/50 [00:07<00:23,  1.57it/s][A

 28%|█████████████████████████▊                                                                  | 14/50 [00:08<00:22,  1.57it/s][A

 30%|███████████████████████████▌                                                                | 15/50 [00:08<00:22,  1.58it/s][A

 32%|█████████████████████████████▍                                                              | 16/50 [00:09<00:21,  1.58it/s][A

 34%|███████████████████████████████▎                                                            | 17/50 [00:10<00:21,  1.54it/s][A

 36%|█████████████████████████████████                                                           | 18/50 [00:10<00:20,  1.55it/s][A

 38%|██████████████████████████████████▉                                                         | 19/50 [00:11<00:19,  1.56it/s][A

 40%|████████████████████████████████████▊                                                       | 20/50 [00:12<00:19,  1.56it/s][A

 42%|██████████████████████████████████████▋                                                     | 21/50 [00:12<00:18,  1.56it/s][A

 44%|████████████████████████████████████████▍                                                   | 22/50 [00:13<00:17,  1.56it/s][A

 46%|██████████████████████████████████████████▎                                                 | 23/50 [00:14<00:17,  1.57it/s][A

 48%|████████████████████████████████████████████▏                                               | 24/50 [00:14<00:16,  1.57it/s][A

 50%|██████████████████████████████████████████████                                              | 25/50 [00:15<00:16,  1.54it/s][A

 52%|███████████████████████████████████████████████▊                                            | 26/50 [00:16<00:15,  1.56it/s][A

 54%|█████████████████████████████████████████████████▋                                          | 27/50 [00:16<00:14,  1.57it/s][A

 56%|███████████████████████████████████████████████████▌                                        | 28/50 [00:17<00:14,  1.57it/s][A

 58%|█████████████████████████████████████████████████████▎                                      | 29/50 [00:17<00:13,  1.57it/s][A

 60%|███████████████████████████████████████████████████████▏                                    | 30/50 [00:18<00:12,  1.57it/s][A

 62%|█████████████████████████████████████████████████████████                                   | 31/50 [00:19<00:12,  1.57it/s][A

 64%|██████████████████████████████████████████████████████████▉                                 | 32/50 [00:19<00:11,  1.58it/s][A

 66%|████████████████████████████████████████████████████████████▋                               | 33/50 [00:20<00:10,  1.55it/s][A

 68%|██████████████████████████████████████████████████████████████▌                             | 34/50 [00:21<00:10,  1.56it/s][A

 70%|████████████████████████████████████████████████████████████████▍                           | 35/50 [00:21<00:09,  1.56it/s][A

 72%|██████████████████████████████████████████████████████████████████▏                         | 36/50 [00:22<00:08,  1.57it/s][A

 74%|████████████████████████████████████████████████████████████████████                        | 37/50 [00:23<00:08,  1.57it/s][A

 76%|█████████████████████████████████████████████████████████████████████▉                      | 38/50 [00:23<00:07,  1.57it/s][A

 78%|███████████████████████████████████████████████████████████████████████▊                    | 39/50 [00:24<00:07,  1.57it/s][A

 80%|█████████████████████████████████████████████████████████████████████████▌                  | 40/50 [00:24<00:06,  1.57it/s][A

 82%|███████████████████████████████████████████████████████████████████████████▍                | 41/50 [00:25<00:05,  1.52it/s][A

 84%|█████████████████████████████████████████████████████████████████████████████▎              | 42/50 [00:26<00:05,  1.55it/s][A

 86%|███████████████████████████████████████████████████████████████████████████████             | 43/50 [00:26<00:04,  1.55it/s][A

 88%|████████████████████████████████████████████████████████████████████████████████▉           | 44/50 [00:27<00:03,  1.56it/s][A

 90%|██████████████████████████████████████████████████████████████████████████████████▊         | 45/50 [00:28<00:03,  1.56it/s][A

 92%|████████████████████████████████████████████████████████████████████████████████████▋       | 46/50 [00:28<00:02,  1.57it/s][A

 94%|██████████████████████████████████████████████████████████████████████████████████████▍     | 47/50 [00:29<00:01,  1.57it/s][A

 96%|████████████████████████████████████████████████████████████████████████████████████████▎   | 48/50 [00:30<00:01,  1.57it/s][A

 98%|██████████████████████████████████████████████████████████████████████████████████████████▏ | 49/50 [00:30<00:00,  1.54it/s][ATraceback (most recent call last):
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
    finalizer()
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
    res = self._callback(*self._args, **self._kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
    rmtree(tempdir, onerror=onerror)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
    _rmtree_safe_fd(stack, onexc)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
    onexc(func, path, err)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
    return onerror(func, path, exc_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
    onexc(os.unlink, fullname, err)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
    return onerror(func, path, exc_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
    os.unlink(entry.name, dir_fd=topfd)
OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-0o4ijedz'
Traceback (most recent call last):
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
    finalizer()
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
    res = self._callback(*self._args, **self._kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
    rmtree(tempdir, onerror=onerror)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
    _rmtree_safe_fd(stack, onexc)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
    onexc(func, path, err)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
    return onerror(func, path, exc_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
    onexc(os.unlink, fullname, err)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
    return onerror(func, path, exc_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
    os.unlink(entry.name, dir_fd=topfd)
OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-_gbgqw_w'
Traceback (most recent call last):
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
    finalizer()
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
    res = self._callback(*self._args, **self._kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
    rmtree(tempdir, onerror=onerror)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
    _rmtree_safe_fd(stack, onexc)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
    onexc(func, path, err)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
    return onerror(func, path, exc_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
    onexc(os.unlink, fullname, err)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
    return onerror(func, path, exc_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
    os.unlink(entry.name, dir_fd=topfd)
OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-tu6wd134'
Traceback (most recent call last):
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
    finalizer()
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
    res = self._callback(*self._args, **self._kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
    rmtree(tempdir, onerror=onerror)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
    _rmtree_safe_fd(stack, onexc)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
    onexc(func, path, err)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
    return onerror(func, path, exc_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
    onexc(os.unlink, fullname, err)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
    return onerror(func, path, exc_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
    os.unlink(entry.name, dir_fd=topfd)
OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-9x0bzt02'


100%|████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:31<00:00,  1.52it/s][A
                                                                                                                                 

                                                                                                                                 
[A{'eval_loss': 0.770285427570343, 'eval_runtime': 32.3968, 'eval_samples_per_second': 12.347, 'eval_steps_per_second': 1.543, 'eval_ppl': 2.16038, 'memory/max_active (GiB)': 42.7, 'memory/max_allocated (GiB)': 42.7, 'memory/device_reserved (GiB)': 61.76, 'epoch': 0.53, 'tokens/train_per_sec_per_gpu': 0.0}

 53%|████████████████████████████████████████████████▍                                           | 30/57 [10:24<07:52, 17.51s/it]

100%|████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:31<00:00,  1.52it/s][A

                                                                                                                                 [A[2026-04-12 22:38:55,542] [INFO] [axolotl.core.trainers.base._save:721] [PID:1912118] Saving model checkpoint to /home/tkwang/scratch/SecSteer-v2/axolotl-outputs/lora/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx3-v2/checkpoint-30
[2026-04-12 22:38:56,497] [WARNING] [py.warnings._showwarnmsg:112] [PID:1912118] /scratch/tkwang/SecSteer-v2/.venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
  return func(*args, **kwargs)


 54%|██████████████████████████████████████████████████                                          | 31/57 [10:43<12:03, 27.84s/it]
                                                                                                                                 
{'loss': 6.1665, 'grad_norm': 0.4019947052001953, 'learning_rate': 2.1207569948445724e-05, 'ppl': 476.51538, 'memory/max_active (GiB)': 46.13, 'memory/max_allocated (GiB)': 46.13, 'memory/device_reserved (GiB)': 61.76, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.54}

 54%|██████████████████████████████████████████████████                                          | 31/57 [10:43<12:03, 27.84s/it]
 56%|███████████████████████████████████████████████████▋                                        | 32/57 [11:00<10:17, 24.69s/it]
                                                                                                                                 
{'loss': 5.619, 'grad_norm': 0.41473397612571716, 'learning_rate': 2e-05, 'ppl': 275.61363, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.76, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.56}

 56%|███████████████████████████████████████████████████▋                                        | 32/57 [11:00<10:17, 24.69s/it]
 58%|█████████████████████████████████████████████████████▎                                      | 33/57 [11:18<09:00, 22.52s/it]
                                                                                                                                 
{'loss': 5.9706, 'grad_norm': 0.3841664493083954, 'learning_rate': 1.879243005155428e-05, 'ppl': 391.74064, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.76, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.58}

 58%|█████████████████████████████████████████████████████▎                                      | 33/57 [11:18<09:00, 22.52s/it]
 60%|██████████████████████████████████████████████████████▉                                     | 34/57 [11:35<08:02, 20.98s/it]
                                                                                                                                 
{'loss': 6.3295, 'grad_norm': 0.3837085962295532, 'learning_rate': 1.758926639489354e-05, 'ppl': 560.87609, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.76, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.6}

 60%|██████████████████████████████████████████████████████▉                                     | 34/57 [11:35<08:02, 20.98s/it]
 61%|████████████████████████████████████████████████████████▍                                   | 35/57 [11:53<07:18, 19.92s/it]
                                                                                                                                 
{'loss': 6.3743, 'grad_norm': 0.3721914291381836, 'learning_rate': 1.6394899243721887e-05, 'ppl': 586.57468, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.76, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.61}

 61%|████████████████████████████████████████████████████████▍                                   | 35/57 [11:53<07:18, 19.92s/it]
 63%|██████████████████████████████████████████████████████████                                  | 36/57 [12:10<06:42, 19.15s/it]
                                                                                                                                 
{'loss': 6.171, 'grad_norm': 0.5033831000328064, 'learning_rate': 1.5213686714248852e-05, 'ppl': 478.66453, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.76, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.63}

 63%|██████████████████████████████████████████████████████████                                  | 36/57 [12:10<06:42, 19.15s/it]
 65%|███████████████████████████████████████████████████████████▋                                | 37/57 [12:28<06:13, 18.66s/it]
                                                                                                                                 
{'loss': 5.9348, 'grad_norm': 0.31661561131477356, 'learning_rate': 1.4049938922895945e-05, 'ppl': 377.9644, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.76, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.65}

 65%|███████████████████████████████████████████████████████████▋                                | 37/57 [12:28<06:13, 18.66s/it]
 67%|█████████████████████████████████████████████████████████████▎                              | 38/57 [12:45<05:47, 18.28s/it]
                                                                                                                                 
{'loss': 6.0972, 'grad_norm': 0.3592473268508911, 'learning_rate': 1.2907902259149287e-05, 'ppl': 444.61111, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.76, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.67}

 67%|█████████████████████████████████████████████████████████████▎                              | 38/57 [12:45<05:47, 18.28s/it]
 68%|██████████████████████████████████████████████████████████████▉                             | 39/57 [13:02<05:23, 18.00s/it]
                                                                                                                                 
{'loss': 6.2305, 'grad_norm': 0.4451168477535248, 'learning_rate': 1.1791743890944869e-05, 'ppl': 508.00942, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.76, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.68}

 68%|██████████████████████████████████████████████████████████████▉                             | 39/57 [13:02<05:23, 18.00s/it]
 70%|████████████████████████████████████████████████████████████████▌                           | 40/57 [13:20<05:02, 17.82s/it]
                                                                                                                                 
{'loss': 6.6532, 'grad_norm': 0.447296679019928, 'learning_rate': 1.070553655912463e-05, 'ppl': 775.2612, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.76, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.7}

 70%|████████████████████████████████████████████████████████████████▌                           | 40/57 [13:20<05:02, 17.82s/it]
 72%|██████████████████████████████████████████████████████████████████▏                         | 41/57 [13:37<04:43, 17.70s/it]
                                                                                                                                 
{'loss': 6.8147, 'grad_norm': 0.3663331866264343, 'learning_rate': 9.653243716446862e-06, 'ppl': 911.14313, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.76, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.72}

 72%|██████████████████████████████████████████████████████████████████▏                         | 41/57 [13:37<04:43, 17.70s/it]
 74%|███████████████████████████████████████████████████████████████████▊                        | 42/57 [13:55<04:24, 17.60s/it]
                                                                                                                                 
{'loss': 6.6086, 'grad_norm': 0.4453074038028717, 'learning_rate': 8.638705065376887e-06, 'ppl': 741.44427, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.76, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.74}

 74%|███████████████████████████████████████████████████████████████████▊                        | 42/57 [13:55<04:24, 17.60s/it]
 75%|█████████████████████████████████████████████████████████████████████▍                      | 43/57 [14:12<04:05, 17.55s/it]
                                                                                                                                 
{'loss': 6.1412, 'grad_norm': 0.34715738892555237, 'learning_rate': 7.665622547429139e-06, 'ppl': 464.61077, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.76, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.75}

 75%|█████████████████████████████████████████████████████████████████████▍                      | 43/57 [14:12<04:05, 17.55s/it]
 77%|███████████████████████████████████████████████████████████████████████                     | 44/57 [14:29<03:47, 17.48s/it]
                                                                                                                                 
{'loss': 6.1923, 'grad_norm': 0.45605599880218506, 'learning_rate': 6.737546835184101e-06, 'ppl': 488.96944, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.76, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.77}

 77%|███████████████████████████████████████████████████████████████████████                     | 44/57 [14:29<03:47, 17.48s/it]
 79%|████████████████████████████████████████████████████████████████████████▋                   | 45/57 [14:47<03:29, 17.46s/it]
                                                                                                                                 
{'loss': 6.3473, 'grad_norm': 0.3942156136035919, 'learning_rate': 5.857864376269051e-06, 'ppl': 570.94906, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.76, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.79}

 79%|████████████████████████████████████████████████████████████████████████▋                   | 45/57 [14:47<03:29, 17.46s/it][2026-04-12 22:43:18,656] [INFO] [axolotl.core.trainers.base.evaluate:400] [PID:1912118] Running evaluation step...


  0%|                                                                                                     | 0/50 [00:00<?, ?it/s][A

  4%|███▋                                                                                         | 2/50 [00:00<00:15,  3.12it/s][A

  6%|█████▌                                                                                       | 3/50 [00:01<00:21,  2.21it/s][A

  8%|███████▍                                                                                     | 4/50 [00:01<00:24,  1.89it/s][A

 10%|█████████▎                                                                                   | 5/50 [00:02<00:25,  1.77it/s][A

 12%|███████████▏                                                                                 | 6/50 [00:03<00:25,  1.70it/s][A

 14%|█████████████                                                                                | 7/50 [00:03<00:25,  1.66it/s][A

 16%|██████████████▉                                                                              | 8/50 [00:04<00:25,  1.63it/s][A

 18%|████████████████▋                                                                            | 9/50 [00:05<00:26,  1.57it/s][A

 20%|██████████████████▍                                                                         | 10/50 [00:05<00:25,  1.58it/s][A

 22%|████████████████████▏                                                                       | 11/50 [00:06<00:24,  1.58it/s][A

 24%|██████████████████████                                                                      | 12/50 [00:07<00:24,  1.57it/s][A

 26%|███████████████████████▉                                                                    | 13/50 [00:07<00:23,  1.57it/s][A

 28%|█████████████████████████▊                                                                  | 14/50 [00:08<00:22,  1.58it/s][A

 30%|███████████████████████████▌                                                                | 15/50 [00:08<00:22,  1.58it/s][A

 32%|█████████████████████████████▍                                                              | 16/50 [00:09<00:21,  1.58it/s][A

 34%|███████████████████████████████▎                                                            | 17/50 [00:10<00:21,  1.55it/s][A

 36%|█████████████████████████████████                                                           | 18/50 [00:10<00:20,  1.56it/s][A

 38%|██████████████████████████████████▉                                                         | 19/50 [00:11<00:19,  1.56it/s][A

 40%|████████████████████████████████████▊                                                       | 20/50 [00:12<00:19,  1.57it/s][A

 42%|██████████████████████████████████████▋                                                     | 21/50 [00:12<00:18,  1.56it/s][A

 44%|████████████████████████████████████████▍                                                   | 22/50 [00:13<00:17,  1.57it/s][A

 46%|██████████████████████████████████████████▎                                                 | 23/50 [00:14<00:17,  1.57it/s][A

 48%|████████████████████████████████████████████▏                                               | 24/50 [00:14<00:16,  1.58it/s][A

 50%|██████████████████████████████████████████████                                              | 25/50 [00:15<00:16,  1.55it/s][A

 52%|███████████████████████████████████████████████▊                                            | 26/50 [00:16<00:15,  1.55it/s][A

 54%|█████████████████████████████████████████████████▋                                          | 27/50 [00:16<00:14,  1.56it/s][A

 56%|███████████████████████████████████████████████████▌                                        | 28/50 [00:17<00:14,  1.57it/s][A

 58%|█████████████████████████████████████████████████████▎                                      | 29/50 [00:17<00:13,  1.56it/s][A

 60%|███████████████████████████████████████████████████████▏                                    | 30/50 [00:18<00:12,  1.57it/s][A

 62%|█████████████████████████████████████████████████████████                                   | 31/50 [00:19<00:12,  1.57it/s][A

 64%|██████████████████████████████████████████████████████████▉                                 | 32/50 [00:19<00:11,  1.57it/s][A

 66%|████████████████████████████████████████████████████████████▋                               | 33/50 [00:20<00:11,  1.54it/s][A

 68%|██████████████████████████████████████████████████████████████▌                             | 34/50 [00:21<00:10,  1.55it/s][A

 70%|████████████████████████████████████████████████████████████████▍                           | 35/50 [00:21<00:09,  1.56it/s][A

 72%|██████████████████████████████████████████████████████████████████▏                         | 36/50 [00:22<00:08,  1.56it/s][A

 74%|████████████████████████████████████████████████████████████████████                        | 37/50 [00:23<00:08,  1.57it/s][A

 76%|█████████████████████████████████████████████████████████████████████▉                      | 38/50 [00:23<00:07,  1.57it/s][A

 78%|███████████████████████████████████████████████████████████████████████▊                    | 39/50 [00:24<00:07,  1.57it/s][A

 80%|█████████████████████████████████████████████████████████████████████████▌                  | 40/50 [00:24<00:06,  1.57it/s][A

 82%|███████████████████████████████████████████████████████████████████████████▍                | 41/50 [00:25<00:05,  1.53it/s][A

 84%|█████████████████████████████████████████████████████████████████████████████▎              | 42/50 [00:26<00:05,  1.55it/s][A

 86%|███████████████████████████████████████████████████████████████████████████████             | 43/50 [00:26<00:04,  1.55it/s][A

 88%|████████████████████████████████████████████████████████████████████████████████▉           | 44/50 [00:27<00:03,  1.56it/s][A

 90%|██████████████████████████████████████████████████████████████████████████████████▊         | 45/50 [00:28<00:03,  1.56it/s][A

 92%|████████████████████████████████████████████████████████████████████████████████████▋       | 46/50 [00:28<00:02,  1.56it/s][A

 94%|██████████████████████████████████████████████████████████████████████████████████████▍     | 47/50 [00:29<00:01,  1.57it/s][A

 96%|████████████████████████████████████████████████████████████████████████████████████████▎   | 48/50 [00:30<00:01,  1.57it/s][A

 98%|██████████████████████████████████████████████████████████████████████████████████████████▏ | 49/50 [00:30<00:00,  1.55it/s][ATraceback (most recent call last):
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
    finalizer()
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
    res = self._callback(*self._args, **self._kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
    rmtree(tempdir, onerror=onerror)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
    _rmtree_safe_fd(stack, onexc)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
    onexc(func, path, err)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
    return onerror(func, path, exc_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
    onexc(os.unlink, fullname, err)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
    return onerror(func, path, exc_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
    os.unlink(entry.name, dir_fd=topfd)
OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-wfxlr8h3'
Traceback (most recent call last):
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
    finalizer()
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
    res = self._callback(*self._args, **self._kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
    rmtree(tempdir, onerror=onerror)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
    _rmtree_safe_fd(stack, onexc)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
    onexc(func, path, err)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
    return onerror(func, path, exc_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
    onexc(os.unlink, fullname, err)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
    return onerror(func, path, exc_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
    os.unlink(entry.name, dir_fd=topfd)
OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-u4imch6o'
Traceback (most recent call last):
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
    finalizer()
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
    res = self._callback(*self._args, **self._kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
    rmtree(tempdir, onerror=onerror)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
    _rmtree_safe_fd(stack, onexc)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
    onexc(func, path, err)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
    return onerror(func, path, exc_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
    onexc(os.unlink, fullname, err)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
    return onerror(func, path, exc_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
    os.unlink(entry.name, dir_fd=topfd)
OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-opagxwej'
Traceback (most recent call last):
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
    finalizer()
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
    res = self._callback(*self._args, **self._kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
    rmtree(tempdir, onerror=onerror)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
    _rmtree_safe_fd(stack, onexc)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
    onexc(func, path, err)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
    return onerror(func, path, exc_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
    onexc(os.unlink, fullname, err)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
    return onerror(func, path, exc_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
    os.unlink(entry.name, dir_fd=topfd)
OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-ny7m792z'


100%|████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:31<00:00,  1.53it/s][A
                                                                                                                                 

                                                                                                                                 
[A{'eval_loss': 0.761513352394104, 'eval_runtime': 32.394, 'eval_samples_per_second': 12.348, 'eval_steps_per_second': 1.543, 'eval_ppl': 2.14151, 'memory/max_active (GiB)': 42.7, 'memory/max_allocated (GiB)': 42.7, 'memory/device_reserved (GiB)': 61.76, 'epoch': 0.79, 'tokens/train_per_sec_per_gpu': 0.0}

 79%|████████████████████████████████████████████████████████████████████████▋                   | 45/57 [15:19<03:29, 17.46s/it]

100%|████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:31<00:00,  1.53it/s][A

                                                                                                                                 [A[2026-04-12 22:43:51,087] [INFO] [axolotl.core.trainers.base._save:721] [PID:1912118] Saving model checkpoint to /home/tkwang/scratch/SecSteer-v2/axolotl-outputs/lora/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx3-v2/checkpoint-45
[2026-04-12 22:43:51,931] [WARNING] [py.warnings._showwarnmsg:112] [PID:1912118] /scratch/tkwang/SecSteer-v2/.venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
  return func(*args, **kwargs)


 81%|██████████████████████████████████████████████████████████████████████████▏                 | 46/57 [15:38<05:04, 27.66s/it]
                                                                                                                                 
{'loss': 5.964, 'grad_norm': 0.3697042167186737, 'learning_rate': 5.029785036577976e-06, 'ppl': 389.16367, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.76, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.81}

 81%|██████████████████████████████████████████████████████████████████████████▏                 | 46/57 [15:38<05:04, 27.66s/it]
 82%|███████████████████████████████████████████████████████████████████████████▊                | 47/57 [15:56<04:05, 24.58s/it]
                                                                                                                                 
{'loss': 6.0951, 'grad_norm': 0.3855403959751129, 'learning_rate': 4.256330387818999e-06, 'ppl': 443.67841, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.76, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.82}

 82%|███████████████████████████████████████████████████████████████████████████▊                | 47/57 [15:56<04:05, 24.58s/it]
 84%|█████████████████████████████████████████████████████████████████████████████▍              | 48/57 [16:13<03:21, 22.41s/it]
                                                                                                                                 
{'loss': 6.5472, 'grad_norm': 0.36347752809524536, 'learning_rate': 3.5403226821268734e-06, 'ppl': 697.28903, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.76, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.84}

 84%|█████████████████████████████████████████████████████████████████████████████▍              | 48/57 [16:13<03:21, 22.41s/it]
 86%|███████████████████████████████████████████████████████████████████████████████             | 49/57 [16:30<02:47, 20.90s/it]
                                                                                                                                 
{'loss': 6.6114, 'grad_norm': 0.40336376428604126, 'learning_rate': 2.8843745539710523e-06, 'ppl': 743.52322, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.76, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.86}

 86%|███████████████████████████████████████████████████████████████████████████████             | 49/57 [16:30<02:47, 20.90s/it]
 88%|████████████████████████████████████████████████████████████████████████████████▋           | 50/57 [16:48<02:19, 19.89s/it]
                                                                                                                                 
{'loss': 5.5329, 'grad_norm': 0.3087441325187683, 'learning_rate': 2.2908794869358044e-06, 'ppl': 252.87619, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.76, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.88}

 88%|████████████████████████████████████████████████████████████████████████████████▋           | 50/57 [16:48<02:19, 19.89s/it]
 89%|██████████████████████████████████████████████████████████████████████████████████▎         | 51/57 [17:05<01:54, 19.12s/it]
                                                                                                                                 
{'loss': 6.3583, 'grad_norm': 0.42559516429901123, 'learning_rate': 1.7620030801581988e-06, 'ppl': 577.26417, 'memory/max_active (GiB)': 46.13, 'memory/max_allocated (GiB)': 46.13, 'memory/device_reserved (GiB)': 61.76, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.89}

 89%|██████████████████████████████████████████████████████████████████████████████████▎         | 51/57 [17:05<01:54, 19.12s/it]
 91%|███████████████████████████████████████████████████████████████████████████████████▉        | 52/57 [17:23<01:33, 18.62s/it]
                                                                                                                                 
{'loss': 5.5182, 'grad_norm': 0.37012147903442383, 'learning_rate': 1.2996751462917057e-06, 'ppl': 249.1861, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.76, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.91}

 91%|███████████████████████████████████████████████████████████████████████████████████▉        | 52/57 [17:23<01:33, 18.62s/it]
 93%|█████████████████████████████████████████████████████████████████████████████████████▌      | 53/57 [17:40<01:12, 18.24s/it]
                                                                                                                                 
{'loss': 5.7734, 'grad_norm': 0.45153290033340454, 'learning_rate': 9.055826698290881e-07, 'ppl': 321.62942, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.76, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.93}

 93%|█████████████████████████████████████████████████████████████████████████████████████▌      | 53/57 [17:40<01:12, 18.24s/it]
 95%|███████████████████████████████████████████████████████████████████████████████████████▏    | 54/57 [17:57<00:54, 18.01s/it]
                                                                                                                                 
{'loss': 5.3825, 'grad_norm': 0.3762929439544678, 'learning_rate': 5.811636514789598e-07, 'ppl': 217.56551, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.76, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.95}

 95%|███████████████████████████████████████████████████████████████████████████████████████▏    | 54/57 [17:57<00:54, 18.01s/it]
 96%|████████████████████████████████████████████████████████████████████████████████████████▊   | 55/57 [18:15<00:35, 17.80s/it]
                                                                                                                                 
{'loss': 6.2599, 'grad_norm': 0.3887091279029846, 'learning_rate': 3.2760186105712964e-07, 'ppl': 523.16662, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.76, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.96}

 96%|████████████████████████████████████████████████████████████████████████████████████████▊   | 55/57 [18:15<00:35, 17.80s/it]
 98%|██████████████████████████████████████████████████████████████████████████████████████████▍ | 56/57 [18:32<00:17, 17.66s/it]
                                                                                                                                 
{'loss': 6.5237, 'grad_norm': 0.45186522603034973, 'learning_rate': 1.4582251803892055e-07, 'ppl': 681.09378, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.76, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.98}

 98%|██████████████████████████████████████████████████████████████████████████████████████████▍ | 56/57 [18:32<00:17, 17.66s/it]Traceback (most recent call last):
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
    finalizer()
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
    res = self._callback(*self._args, **self._kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
    rmtree(tempdir, onerror=onerror)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
    _rmtree_safe_fd(stack, onexc)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
    onexc(func, path, err)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
    return onerror(func, path, exc_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
    onexc(os.unlink, fullname, err)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
    return onerror(func, path, exc_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
    os.unlink(entry.name, dir_fd=topfd)
OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-7slk8rfc'
Traceback (most recent call last):
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
    finalizer()
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
    res = self._callback(*self._args, **self._kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
    rmtree(tempdir, onerror=onerror)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
    _rmtree_safe_fd(stack, onexc)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
    onexc(func, path, err)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
    return onerror(func, path, exc_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
    onexc(os.unlink, fullname, err)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
    return onerror(func, path, exc_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
    os.unlink(entry.name, dir_fd=topfd)
OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-58ctqgen'
Traceback (most recent call last):
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
    finalizer()
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
    res = self._callback(*self._args, **self._kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
    rmtree(tempdir, onerror=onerror)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
    _rmtree_safe_fd(stack, onexc)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
    onexc(func, path, err)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
    return onerror(func, path, exc_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
    onexc(os.unlink, fullname, err)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
    return onerror(func, path, exc_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
    os.unlink(entry.name, dir_fd=topfd)
OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-2ulklgi9'
Traceback (most recent call last):
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
    finalizer()
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
    res = self._callback(*self._args, **self._kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
    rmtree(tempdir, onerror=onerror)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
    _rmtree_safe_fd(stack, onexc)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
    onexc(func, path, err)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
    return onerror(func, path, exc_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
    onexc(os.unlink, fullname, err)
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
    return onerror(func, path, exc_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
    os.unlink(entry.name, dir_fd=topfd)
OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-vsw1snmb'

100%|████████████████████████████████████████████████████████████████████████████████████████████| 57/57 [18:50<00:00, 17.62s/it]
                                                                                                                                 
{'loss': 5.3191, 'grad_norm': 0.37680739164352417, 'learning_rate': 3.648891553365008e-08, 'ppl': 204.20002, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.76, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 1.0}

100%|████████████████████████████████████████████████████████████████████████████████████████████| 57/57 [18:50<00:00, 17.62s/it][2026-04-12 22:47:21,572] [INFO] [axolotl.core.trainers.base._save:721] [PID:1912118] Saving model checkpoint to /home/tkwang/scratch/SecSteer-v2/axolotl-outputs/lora/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx3-v2/checkpoint-57
[2026-04-12 22:47:22,356] [WARNING] [py.warnings._showwarnmsg:112] [PID:1912118] /scratch/tkwang/SecSteer-v2/.venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
  return func(*args, **kwargs)


                                                                                                                                 
{'train_runtime': 1135.4865, 'train_samples_per_second': 3.213, 'train_steps_per_second': 0.05, 'train_loss': 6.3918650610405106, 'memory/max_active (GiB)': 15.01, 'memory/max_allocated (GiB)': 15.01, 'memory/device_reserved (GiB)': 61.76, 'epoch': 1.0, 'tokens/train_per_sec_per_gpu': 0.0}

100%|████████████████████████████████████████████████████████████████████████████████████████████| 57/57 [18:51<00:00, 17.62s/it]
100%|████████████████████████████████████████████████████████████████████████████████████████████| 57/57 [18:51<00:00, 19.86s/it]
[2026-04-12 22:47:23,346] [INFO] [axolotl.train.save_trained_model:233] [PID:1912118] Training completed! Saving trained model to /home/tkwang/scratch/SecSteer-v2/axolotl-outputs/lora/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx3-v2.
[2026-04-12 22:47:23,851] [INFO] [axolotl.train.save_trained_model:351] [PID:1912118] Model successfully saved to /home/tkwang/scratch/SecSteer-v2/axolotl-outputs/lora/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx3-v2