Instructions to use felixwangg/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx0-v2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use felixwangg/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx0-v2 with PEFT:

from peft import PeftModel
from transformers import AutoModelForCausalLM

base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-7B-Instruct")
model = PeftModel.from_pretrained(base_model, "felixwangg/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx0-v2")

Transformers

How to use felixwangg/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx0-v2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="felixwangg/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx0-v2")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("felixwangg/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx0-v2")
model = AutoModelForCausalLM.from_pretrained("felixwangg/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx0-v2", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use felixwangg/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx0-v2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "felixwangg/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx0-v2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "felixwangg/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx0-v2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/felixwangg/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx0-v2

SGLang

How to use felixwangg/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx0-v2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "felixwangg/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx0-v2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "felixwangg/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx0-v2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "felixwangg/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx0-v2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "felixwangg/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx0-v2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use felixwangg/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx0-v2 with Docker Model Runner:
```
docker model run hf.co/felixwangg/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx0-v2
```

Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx0-v2 / debug.log

felixwangg

Upload folder using huggingface_hub

c2d9810 verified 4 months ago

Raw

History Blame Contribute Delete

157 kB

	[2026-04-12 22:06:37,755] [DEBUG] [axolotl.utils.config.log_gpu_memory_usage:127] [PID:1885930] baseline 0.000GB ()
	[2026-04-12 22:06:37,755] [INFO] [axolotl.cli.config.load_cfg:259] [PID:1885930] config:
	{
	"activation_offloading": false,
	"adapter": "lora",
	"axolotl_config_path": "./axolotl_configs/Qwen2.5-coder-7b-instruct/primevul/lora-primevul-plus-token-diff-mask-ctx0.yaml",
	"base_model": "Qwen/Qwen2.5-Coder-7B-Instruct",
	"base_model_config": "Qwen/Qwen2.5-Coder-7B-Instruct",
	"batch_size": 64,
	"bf16": true,
	"capabilities": {
	"bf16": true,
	"compute_capability": "sm_90",
	"fp8": true,
	"n_gpu": 2,
	"n_node": 1
	},
	"context_parallel_size": 1,
	"dataloader_num_workers": 2,
	"dataloader_pin_memory": true,
	"dataloader_prefetch_factor": 256,
	"dataset_num_proc": 112,
	"dataset_prepared_path": "/home/tkwang/scratch/SecSteer-v2/axolotl-datasets/lora/Qwen2.5-Coder-7B/prime_vul_plus_splitted_token_diff_mask_skip_indent_ctx0_chat_v2",
	"datasets": [
	{
	"message_property_mappings": {
	"content": "content",
	"role": "role"
	},
	"path": "felixwangg/prime_vul_plus_splitted_token_diff_mask_skip_indent_ctx0_chat_v2",
	"split": "train",
	"trust_remote_code": false,
	"type": "pretokenized"
	}
	],
	"ddp": true,
	"device": "cuda:0",
	"device_map": {
	"": 0
	},
	"dion_rank_fraction": 1.0,
	"dion_rank_multiple_of": 1,
	"early_stopping_patience": 1000,
	"env_capabilities": {
	"torch_version": "2.10.0"
	},
	"eval_batch_size": 4,
	"eval_causal_lm_metrics": [
	"sacrebleu",
	"comet",
	"ter",
	"chrf"
	],
	"eval_max_new_tokens": 128,
	"eval_sample_packing": false,
	"eval_steps": 15,
	"eval_table_size": 0,
	"experimental_skip_move_to_device": true,
	"flash_attention": true,
	"fp16": false,
	"gradient_accumulation_steps": 8,
	"gradient_checkpointing": true,
	"gradient_checkpointing_kwargs": {
	"use_reentrant": true
	},
	"include_tkps": true,
	"is_falcon_derived_model": false,
	"is_llama_derived_model": false,
	"is_mistral_derived_model": false,
	"learning_rate": 4e-05,
	"lisa_layers_attribute": "model.layers",
	"load_best_model_at_end": true,
	"load_in_4bit": false,
	"load_in_8bit": false,
	"local_rank": 0,
	"logging_steps": 1,
	"lora_alpha": 16,
	"lora_dropout": 0.05,
	"lora_r": 16,
	"lora_target_linear": true,
	"loraplus_lr_embedding": 1e-06,
	"lr_scheduler": "cosine",
	"mean_resizing_embeddings": false,
	"merge_lora": true,
	"micro_batch_size": 4,
	"model_config_type": "qwen2",
	"num_epochs": 1.0,
	"optimizer": "adamw_torch",
	"otel_metrics_host": "localhost",
	"otel_metrics_port": 8000,
	"output_dir": "/home/tkwang/scratch/SecSteer-v2/axolotl-outputs/lora/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx0-v2",
	"pad_to_sequence_len": true,
	"plugins": [
	"diff_mask_trainer.plugin.DiffMaskPlugin"
	],
	"pretrain_multipack_attn": true,
	"profiler_steps_start": 0,
	"qlora_sharded_model_loading": false,
	"ray_num_workers": 1,
	"resources_per_worker": {
	"GPU": 1
	},
	"sample_packing": false,
	"sample_packing_bin_size": 200,
	"sample_packing_group_size": 100000,
	"save_only_model": false,
	"save_safetensors": true,
	"save_steps": 15,
	"save_total_limit": 1000,
	"sequence_len": 4096,
	"shuffle_before_merging_datasets": false,
	"shuffle_merged_datasets": true,
	"skip_prepare_dataset": false,
	"streaming_multipack_buffer_size": 10000,
	"strict": false,
	"tensor_parallel_size": 1,
	"test_datasets": [
	{
	"message_property_mappings": {
	"content": "content",
	"role": "role"
	},
	"path": "felixwangg/prime_vul_plus_splitted_token_diff_mask_skip_indent_ctx0_chat_v2",
	"split": "validation",
	"trust_remote_code": false,
	"type": "pretokenized"
	}
	],
	"tf32": false,
	"tiled_mlp_use_original_mlp": true,
	"tokenizer_config": "Qwen/Qwen2.5-Coder-7B-Instruct",
	"tokenizer_save_jinja_files": true,
	"tokenizer_type": "AutoTokenizer",
	"torch_dtype": "torch.bfloat16",
	"train_on_inputs": false,
	"trl": {
	"log_completions": false,
	"mask_truncated_completions": false,
	"ref_model_mixup_alpha": 0.9,
	"ref_model_sync_steps": 64,
	"scale_rewards": true,
	"sync_ref_model": false,
	"use_vllm": false,
	"vllm_server_host": "0.0.0.0",
	"vllm_server_port": 8000
	},
	"type_of_model": "Qwen2ForCausalLM",
	"use_otel_metrics": false,
	"use_ray": false,
	"use_wandb": true,
	"val_set_size": 0.0,
	"vllm": {
	"device": "auto",
	"dtype": "auto",
	"gpu_memory_utilization": 0.9,
	"host": "0.0.0.0",
	"port": 8000
	},
	"wandb_entity": "wtkuan",
	"wandb_log_model": "false",
	"wandb_name": "Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx0-v2",
	"wandb_project": "diff-mask-sft-primevul-ctx-0",
	"wandb_watch": "false",
	"warmup_ratio": 0.1,
	"weight_decay": 0.02,
	"world_size": 2
	}
	[2026-04-12 22:06:38,353] [DEBUG] [axolotl.loaders.tokenizer.load_tokenizer:285] [PID:1885930] EOS: 151645 / <\|im_end\|>
	[2026-04-12 22:06:38,353] [DEBUG] [axolotl.loaders.tokenizer.load_tokenizer:286] [PID:1885930] BOS: None / None
	[2026-04-12 22:06:38,353] [DEBUG] [axolotl.loaders.tokenizer.load_tokenizer:287] [PID:1885930] PAD: 151643 / <\|endoftext\|>
	[2026-04-12 22:06:38,353] [DEBUG] [axolotl.loaders.tokenizer.load_tokenizer:288] [PID:1885930] UNK: None / None
	[2026-04-12 22:06:38,497] [INFO] [axolotl.utils.data.shared.load_preprocessed_dataset:475] [PID:1885930] Loading prepared dataset from disk at /home/tkwang/scratch/SecSteer-v2/axolotl-datasets/lora/Qwen2.5-Coder-7B/prime_vul_plus_splitted_token_diff_mask_skip_indent_ctx0_chat_v2/dfd1c42d292b0e1ab5ed322b4783f61f...
	[2026-04-12 22:06:38,514] [INFO] [axolotl.utils.data.shared.load_preprocessed_dataset:475] [PID:1885930] Loading prepared dataset from disk at /home/tkwang/scratch/SecSteer-v2/axolotl-datasets/lora/Qwen2.5-Coder-7B/prime_vul_plus_splitted_token_diff_mask_skip_indent_ctx0_chat_v2/c7eaec7bad218ebeb659420ad3e9b663...
	[2026-04-12 22:06:38,576] [DEBUG] [axolotl.utils.trainer.calculate_total_num_steps:417] [PID:1885930] total_num_tokens: 3_735_150
	[2026-04-12 22:06:38,917] [DEBUG] [axolotl.utils.trainer.calculate_total_num_steps:435] [PID:1885930] `total_supervised_tokens: 2_946_329`
	[2026-04-12 22:06:38,917] [DEBUG] [axolotl.utils.trainer.calculate_total_num_steps:533] [PID:1885930] total_num_steps: 57
	[2026-04-12 22:06:38,917] [INFO] [axolotl.utils.data.sft._prepare_standard_dataset:121] [PID:1885930] Maximum number of steps set at 57
	[2026-04-12 22:06:38,937] [DEBUG] [axolotl.train.setup_model_and_tokenizer:70] [PID:1885930] loading tokenizer... Qwen/Qwen2.5-Coder-7B-Instruct
	[2026-04-12 22:06:39,412] [DEBUG] [axolotl.loaders.tokenizer.load_tokenizer:285] [PID:1885930] EOS: 151645 / <\|im_end\|>
	[2026-04-12 22:06:39,412] [DEBUG] [axolotl.loaders.tokenizer.load_tokenizer:286] [PID:1885930] BOS: None / None
	[2026-04-12 22:06:39,412] [DEBUG] [axolotl.loaders.tokenizer.load_tokenizer:287] [PID:1885930] PAD: 151643 / <\|endoftext\|>
	[2026-04-12 22:06:39,412] [DEBUG] [axolotl.loaders.tokenizer.load_tokenizer:288] [PID:1885930] UNK: None / None
	[2026-04-12 22:06:39,412] [DEBUG] [axolotl.train.setup_model_and_tokenizer:82] [PID:1885930] Loading model
	[2026-04-12 22:06:39,500] [DEBUG] [axolotl.monkeypatch.transformers.trainer_loss_calc.patch_evaluation_loop:87] [PID:1885930] Patched Trainer.evaluation_loop with nanmean loss calculation
	[2026-04-12 22:06:39,501] [DEBUG] [axolotl.monkeypatch.transformers.trainer_loss_calc.patch_maybe_log_save_evaluate:138] [PID:1885930] Patched Trainer._maybe_log_save_evaluate with nanmean loss calculation
	Loading checkpoint shards: 0%\| \| 0/4 [00:00<?, ?it/s] Loading checkpoint shards: 0%\| \| 0/4 [00:00<?, ?it/s] Loading checkpoint shards: 25%\|████████████████▊ \| 1/4 [00:00<00:02, 1.18it/s] Loading checkpoint shards: 25%\|████████████████▊ \| 1/4 [00:00<00:02, 1.23it/s] Loading checkpoint shards: 50%\|█████████████████████████████████▌ \| 2/4 [00:01<00:01, 1.13it/s] Loading checkpoint shards: 50%\|█████████████████████████████████▌ \| 2/4 [00:01<00:01, 1.11it/s] Loading checkpoint shards: 75%\|██████████████████████████████████████████████████▎ \| 3/4 [00:02<00:00, 1.17it/s] Loading checkpoint shards: 75%\|██████████████████████████████████████████████████▎ \| 3/4 [00:02<00:00, 1.16it/s] Loading checkpoint shards: 100%\|███████████████████████████████████████████████████████████████████\| 4/4 [00:02<00:00, 1.67it/s] Loading checkpoint shards: 100%\|███████████████████████████████████████████████████████████████████\| 4/4 [00:02<00:00, 1.43it/s] Loading checkpoint shards: 100%\|███████████████████████████████████████████████████████████████████\| 4/4 [00:02<00:00, 1.68it/s] Loading checkpoint shards: 100%\|███████████████████████████████████████████████████████████████████\| 4/4 [00:02<00:00, 1.44it/s]
	[2026-04-12 22:06:43,152] [INFO] [axolotl.loaders.model._configure_embedding_dtypes:347] [PID:1885930] Converting modules to torch.bfloat16
	[2026-04-12 22:06:43,154] [DEBUG] [axolotl.loaders.model.log_gpu_memory_usage:127] [PID:1885930] Memory usage after model load 17.233GB (+17.233GB allocated, +18.252GB reserved)
	[2026-04-12 22:06:43,154] [INFO] [axolotl.loaders.adapter.load_lora:81] [PID:1885930] found linear modules: ['down_proj', 'gate_proj', 'k_proj', 'o_proj', 'q_proj', 'up_proj', 'v_proj']
	trainable params: 40,370,176 \|\| all params: 7,655,986,688 \|\| trainable%: 0.5273
	[2026-04-12 22:06:43,464] [DEBUG] [axolotl.loaders.model.log_gpu_memory_usage:127] [PID:1885930] after adapters 14.337GB (+14.337GB allocated, +18.330GB reserved)
	[2026-04-12 22:06:46,455] [WARNING] [py.warnings._showwarnmsg:112] [PID:1885930] /scratch/tkwang/SecSteer-v2/.venv/lib/python3.12/site-packages/trl/extras/vllm_client.py:37: UserWarning: TRL currently supports vLLM versions: 0.10.2, 0.11.0, 0.11.1, 0.11.2, 0.12.0. You have version 0.19.0 installed. We recommend installing a supported version to avoid compatibility issues.
	if is_vllm_available():


	[2026-04-12 22:06:46,456] [WARNING] [py.warnings._showwarnmsg:112] [PID:1885931] /scratch/tkwang/SecSteer-v2/.venv/lib/python3.12/site-packages/trl/extras/vllm_client.py:37: UserWarning: TRL currently supports vLLM versions: 0.10.2, 0.11.0, 0.11.1, 0.11.2, 0.12.0. You have version 0.19.0 installed. We recommend installing a supported version to avoid compatibility issues.
	if is_vllm_available():

	[2026-04-12 22:06:46,690] [WARNING] [py.warnings._showwarnmsg:112] [PID:1885930] /scratch/tkwang/SecSteer-v2/.venv/lib/python3.12/site-packages/trl/trainer/grpo_trainer.py:105: UserWarning: TRL currently supports vLLM versions: 0.10.2, 0.11.0, 0.11.1, 0.11.2, 0.12.0. You have version 0.19.0 installed. We recommend installing a supported version to avoid compatibility issues.
	if is_vllm_available():

	[2026-04-12 22:06:46,690] [WARNING] [py.warnings._showwarnmsg:112] [PID:1885931] /scratch/tkwang/SecSteer-v2/.venv/lib/python3.12/site-packages/trl/trainer/grpo_trainer.py:105: UserWarning: TRL currently supports vLLM versions: 0.10.2, 0.11.0, 0.11.1, 0.11.2, 0.12.0. You have version 0.19.0 installed. We recommend installing a supported version to avoid compatibility issues.
	if is_vllm_available():

	DiffMaskPlugin: patching trainer with alpha=1.0
	DiffMaskPlugin: compute_loss and prediction_step patched
	[2026-04-12 22:06:54,210] [INFO] [axolotl.train.save_initial_configs:413] [PID:1885930] Pre-saving adapter config to /home/tkwang/scratch/SecSteer-v2/axolotl-outputs/lora/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx0-v2...
	[2026-04-12 22:06:54,218] [INFO] [axolotl.train.save_initial_configs:417] [PID:1885930] Pre-saving tokenizer to /home/tkwang/scratch/SecSteer-v2/axolotl-outputs/lora/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx0-v2...
	[2026-04-12 22:06:54,432] [INFO] [axolotl.train.save_initial_configs:422] [PID:1885930] Pre-saving model config to /home/tkwang/scratch/SecSteer-v2/axolotl-outputs/lora/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx0-v2...
	[2026-04-12 22:06:54,447] [INFO] [axolotl.train.execute_training:212] [PID:1885930] Starting trainer...
	DiffMaskPlugin: patching trainer with alpha=1.0
	DiffMaskPlugin: compute_loss and prediction_step patched
	[34m[1mwandb[0m: [wandb.login()] Loaded credentials for https://api.wandb.ai from /home/tkwang/.netrc.
	[34m[1mwandb[0m: Currently logged in as: [33mwtkuan[0m to [32mhttps://api.wandb.ai[0m. Use [1m`wandb login --relogin`[0m to force relogin
	[34m[1mwandb[0m: [38;5;178m⢿[0m Waiting for wandb.init()...
	[Am[2K [34m[1mwandb[0m: [38;5;178m⣻[0m Waiting for wandb.init()...
	[Am[2K [34m[1mwandb[0m: Tracking run with wandb version 0.24.0
	[34m[1mwandb[0m: Run data is saved locally in [35m[1m/scratch/tkwang/SecSteer-v2/wandb/run-20260412_220656-ccuy8vz6[0m
	[34m[1mwandb[0m: Run [1m`wandb offline`[0m to turn off syncing.
	[34m[1mwandb[0m: Syncing run [33mQwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx0-v2[0m
	[34m[1mwandb[0m: ⭐️ View project at [34m[4mhttps://wandb.ai/wtkuan/diff-mask-sft-primevul-ctx-0[0m
	[34m[1mwandb[0m: 🚀 View run at [34m[4mhttps://wandb.ai/wtkuan/diff-mask-sft-primevul-ctx-0/runs/ccuy8vz6[0m
	[34m[1mwandb[0m: Detected [huggingface_hub.inference, mcp, openai] in use.
	[34m[1mwandb[0m: Use W&B Weave for improved LLM call tracing. Install Weave with `pip install weave` then add `import weave` to the top of your script.
	[34m[1mwandb[0m: For more information, check out the docs at: https://weave-docs.wandb.ai/
	[34m[1mwandb[0m: [33mWARNING[0m Saving files without folders. If you want to preserve subdirectories pass base_path to wandb.save, i.e. wandb.save("/mnt/folder/file.h5", base_path="/mnt")
	[34m[1mwandb[0m: [33mWARNING[0m Symlinked 1 file into the W&B run directory; call wandb.save again to sync new files.
	[2026-04-12 22:06:58,545] [INFO] [axolotl.utils.callbacks.on_train_begin:757] [PID:1885930] The Axolotl config has been saved to the WandB run under files.
	0%\| \| 0/57 [00:00<?, ?it/s][2026-04-12 22:06:58,547] [INFO] [axolotl.core.trainers.base.evaluate:400] [PID:1885930] Running evaluation step...

	0%\| \| 0/50 [00:00<?, ?it/s][A
	4%\|███▋ \| 2/50 [00:00<00:13, 3.56it/s][A
	6%\|█████▌ \| 3/50 [00:01<00:20, 2.32it/s][A
	8%\|███████▍ \| 4/50 [00:01<00:23, 1.97it/s][A
	10%\|█████████▎ \| 5/50 [00:02<00:24, 1.82it/s][A
	12%\|███████████▏ \| 6/50 [00:03<00:25, 1.73it/s][A
	14%\|█████████████ \| 7/50 [00:03<00:25, 1.67it/s][A
	16%\|██████████████▉ \| 8/50 [00:04<00:25, 1.64it/s][A
	18%\|████████████████▋ \| 9/50 [00:05<00:25, 1.59it/s][A
	20%\|██████████████████▍ \| 10/50 [00:05<00:25, 1.60it/s][A
	22%\|████████████████████▏ \| 11/50 [00:06<00:24, 1.59it/s][A
	24%\|██████████████████████ \| 12/50 [00:06<00:23, 1.59it/s][A
	26%\|███████████████████████▉ \| 13/50 [00:07<00:23, 1.59it/s][A
	28%\|█████████████████████████▊ \| 14/50 [00:08<00:22, 1.58it/s][A
	30%\|███████████████████████████▌ \| 15/50 [00:08<00:22, 1.58it/s][A
	32%\|█████████████████████████████▍ \| 16/50 [00:09<00:21, 1.57it/s][A
	34%\|███████████████████████████████▎ \| 17/50 [00:10<00:21, 1.55it/s][A
	36%\|█████████████████████████████████ \| 18/50 [00:10<00:20, 1.56it/s][A
	38%\|██████████████████████████████████▉ \| 19/50 [00:11<00:19, 1.56it/s][A
	40%\|████████████████████████████████████▊ \| 20/50 [00:12<00:19, 1.56it/s][A
	42%\|██████████████████████████████████████▋ \| 21/50 [00:12<00:18, 1.56it/s][A
	44%\|████████████████████████████████████████▍ \| 22/50 [00:13<00:17, 1.57it/s][A
	46%\|██████████████████████████████████████████▎ \| 23/50 [00:13<00:17, 1.57it/s][A
	48%\|████████████████████████████████████████████▏ \| 24/50 [00:14<00:16, 1.57it/s][A
	50%\|██████████████████████████████████████████████ \| 25/50 [00:15<00:16, 1.54it/s][A
	52%\|███████████████████████████████████████████████▊ \| 26/50 [00:15<00:15, 1.55it/s][A
	54%\|█████████████████████████████████████████████████▋ \| 27/50 [00:16<00:14, 1.55it/s][A
	56%\|███████████████████████████████████████████████████▌ \| 28/50 [00:17<00:14, 1.55it/s][A
	58%\|█████████████████████████████████████████████████████▎ \| 29/50 [00:17<00:13, 1.56it/s][A
	60%\|███████████████████████████████████████████████████████▏ \| 30/50 [00:18<00:12, 1.56it/s][A
	62%\|█████████████████████████████████████████████████████████ \| 31/50 [00:19<00:12, 1.57it/s][A
	64%\|██████████████████████████████████████████████████████████▉ \| 32/50 [00:19<00:11, 1.57it/s][A
	66%\|████████████████████████████████████████████████████████████▋ \| 33/50 [00:20<00:11, 1.54it/s][A
	68%\|██████████████████████████████████████████████████████████████▌ \| 34/50 [00:21<00:10, 1.55it/s][A
	70%\|████████████████████████████████████████████████████████████████▍ \| 35/50 [00:21<00:09, 1.55it/s][A
	72%\|██████████████████████████████████████████████████████████████████▏ \| 36/50 [00:22<00:08, 1.56it/s][A
	74%\|████████████████████████████████████████████████████████████████████ \| 37/50 [00:22<00:08, 1.55it/s][A
	76%\|█████████████████████████████████████████████████████████████████████▉ \| 38/50 [00:23<00:07, 1.56it/s][A
	78%\|███████████████████████████████████████████████████████████████████████▊ \| 39/50 [00:24<00:07, 1.56it/s][A
	80%\|█████████████████████████████████████████████████████████████████████████▌ \| 40/50 [00:24<00:06, 1.57it/s][A
	82%\|███████████████████████████████████████████████████████████████████████████▍ \| 41/50 [00:25<00:05, 1.54it/s][A
	84%\|█████████████████████████████████████████████████████████████████████████████▎ \| 42/50 [00:26<00:05, 1.55it/s][A
	86%\|███████████████████████████████████████████████████████████████████████████████ \| 43/50 [00:26<00:04, 1.56it/s][A
	88%\|████████████████████████████████████████████████████████████████████████████████▉ \| 44/50 [00:27<00:03, 1.56it/s][A
	90%\|██████████████████████████████████████████████████████████████████████████████████▊ \| 45/50 [00:28<00:03, 1.56it/s][A
	92%\|████████████████████████████████████████████████████████████████████████████████████▋ \| 46/50 [00:28<00:02, 1.56it/s][A
	94%\|██████████████████████████████████████████████████████████████████████████████████████▍ \| 47/50 [00:29<00:01, 1.56it/s][A
	96%\|████████████████████████████████████████████████████████████████████████████████████████▎ \| 48/50 [00:30<00:01, 1.56it/s][A
	98%\|██████████████████████████████████████████████████████████████████████████████████████████▏ \| 49/50 [00:30<00:00, 1.53it/s][ATraceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-ity5t2lj'
	Traceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-s0g98ru4'
	Traceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-kh9tko64'
	Traceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-1ncqh8kf'

	100%\|████████████████████████████████████████████████████████████████████████████████████████████\| 50/50 [00:31<00:00, 1.50it/s][A
	[A{'eval_loss': 0.8681871294975281, 'eval_runtime': 32.6781, 'eval_samples_per_second': 12.241, 'eval_steps_per_second': 1.53, 'eval_ppl': 2.38259, 'memory/max_active (GiB)': 42.36, 'memory/max_allocated (GiB)': 42.36, 'memory/device_reserved (GiB)': 51.15, 'epoch': 0}
	0%\| \| 0/57 [00:32<?, ?it/s]
	100%\|████████████████████████████████████████████████████████████████████████████████████████████\| 50/50 [00:31<00:00, 1.50it/s][A
	[A 2%\|█▋ \| 1/57 [00:53<49:45, 53.32s/it] {'loss': 7.2763, 'grad_norm': 0.4794088900089264, 'learning_rate': 0.0, 'ppl': 1445.62929, 'memory/max_active (GiB)': 45.83, 'memory/max_allocated (GiB)': 45.83, 'memory/device_reserved (GiB)': 60.58, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.02}
	2%\|█▋ \| 1/57 [00:53<49:45, 53.32s/it] 4%\|███▎ \| 2/57 [01:10<29:30, 32.18s/it] {'loss': 7.1752, 'grad_norm': 0.474256694316864, 'learning_rate': 8.000000000000001e-06, 'ppl': 1306.6214, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 60.61, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.04}
	4%\|███▎ \| 2/57 [01:10<29:30, 32.18s/it] 5%\|████▉ \| 3/57 [01:28<22:51, 25.40s/it] {'loss': 6.8168, 'grad_norm': 0.5172702670097351, 'learning_rate': 1.6000000000000003e-05, 'ppl': 913.05854, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 60.61, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.05}
	5%\|████▉ \| 3/57 [01:28<22:51, 25.40s/it] 7%\|██████▌ \| 4/57 [01:45<19:39, 22.26s/it] {'loss': 6.7764, 'grad_norm': 0.4488540291786194, 'learning_rate': 2.4e-05, 'ppl': 876.90617, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 60.61, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.07}
	7%\|██████▌ \| 4/57 [01:45<19:39, 22.26s/it] 9%\|████████▏ \| 5/57 [02:02<17:45, 20.50s/it] {'loss': 6.9667, 'grad_norm': 0.5024597644805908, 'learning_rate': 3.2000000000000005e-05, 'ppl': 1060.7166, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 60.61, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.09}
	9%\|████████▏ \| 5/57 [02:02<17:45, 20.50s/it] 11%\|█████████▊ \| 6/57 [02:20<16:30, 19.42s/it] {'loss': 7.7777, 'grad_norm': 0.642304539680481, 'learning_rate': 4e-05, 'ppl': 2386.77891, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 60.61, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.11}
	11%\|█████████▊ \| 6/57 [02:20<16:30, 19.42s/it] 12%\|███████████▍ \| 7/57 [02:37<15:36, 18.73s/it] {'loss': 6.729, 'grad_norm': 0.5520005226135254, 'learning_rate': 3.996351108446635e-05, 'ppl': 836.31054, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 60.61, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.12}
	12%\|███████████▍ \| 7/57 [02:37<15:36, 18.73s/it] 14%\|█████████████ \| 8/57 [02:54<14:57, 18.31s/it] {'loss': 6.9707, 'grad_norm': 0.5780239701271057, 'learning_rate': 3.985417748196108e-05, 'ppl': 1064.96797, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 60.61, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.14}
	14%\|█████████████ \| 8/57 [02:54<14:57, 18.31s/it] 16%\|██████████████▋ \| 9/57 [03:12<14:24, 18.02s/it] {'loss': 6.5023, 'grad_norm': 0.5996211767196655, 'learning_rate': 3.967239813894288e-05, 'ppl': 666.67322, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 60.61, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.16}
	16%\|██████████████▋ \| 9/57 [03:12<14:24, 18.02s/it] 18%\|████████████████▏ \| 10/57 [03:29<13:59, 17.86s/it] {'loss': 6.1406, 'grad_norm': 0.6215760111808777, 'learning_rate': 3.9418836348521045e-05, 'ppl': 464.33209, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 60.61, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.18}
	18%\|████████████████▏ \| 10/57 [03:29<13:59, 17.86s/it] 19%\|█████████████████▊ \| 11/57 [03:47<13:36, 17.76s/it] {'loss': 6.8643, 'grad_norm': 0.6392199993133545, 'learning_rate': 3.909441733017092e-05, 'ppl': 957.47537, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 60.61, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.19}
	19%\|█████████████████▊ \| 11/57 [03:47<13:36, 17.76s/it] 21%\|███████████████████▎ \| 12/57 [04:04<13:14, 17.66s/it] {'loss': 6.7031, 'grad_norm': 0.7280862331390381, 'learning_rate': 3.8700324853708304e-05, 'ppl': 814.92819, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 60.61, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.21}
	21%\|███████████████████▎ \| 12/57 [04:04<13:14, 17.66s/it] 23%\|████████████████████▉ \| 13/57 [04:22<12:52, 17.56s/it] {'loss': 6.6227, 'grad_norm': 0.8180727362632751, 'learning_rate': 3.82379969198418e-05, 'ppl': 751.97268, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 60.61, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.23}
	23%\|████████████████████▉ \| 13/57 [04:22<12:52, 17.56s/it] 25%\|██████████████████████▌ \| 14/57 [04:39<12:34, 17.54s/it] {'loss': 6.5021, 'grad_norm': 0.6895028352737427, 'learning_rate': 3.7709120513064196e-05, 'ppl': 666.5399, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 60.61, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.25}
	25%\|██████████████████████▌ \| 14/57 [04:39<12:34, 17.54s/it] 26%\|████████████████████████▏ \| 15/57 [04:56<12:13, 17.46s/it] {'loss': 7.1944, 'grad_norm': 0.7488486766815186, 'learning_rate': 3.711562544602895e-05, 'ppl': 1331.95092, 'memory/max_active (GiB)': 46.13, 'memory/max_allocated (GiB)': 46.13, 'memory/device_reserved (GiB)': 60.61, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.26}
	26%\|████████████████████████▏ \| 15/57 [04:56<12:13, 17.46s/it][2026-04-12 22:11:55,394] [INFO] [axolotl.core.trainers.base.evaluate:400] [PID:1885930] Running evaluation step...

	0%\| \| 0/50 [00:00<?, ?it/s][A
	4%\|███▋ \| 2/50 [00:00<00:15, 3.19it/s][A
	6%\|█████▌ \| 3/50 [00:01<00:21, 2.21it/s][A
	8%\|███████▍ \| 4/50 [00:01<00:23, 1.92it/s][A
	10%\|█████████▎ \| 5/50 [00:02<00:25, 1.79it/s][A
	12%\|███████████▏ \| 6/50 [00:03<00:25, 1.71it/s][A
	14%\|█████████████ \| 7/50 [00:03<00:25, 1.65it/s][A
	16%\|██████████████▉ \| 8/50 [00:04<00:25, 1.63it/s][A
	18%\|████████████████▋ \| 9/50 [00:05<00:25, 1.58it/s][A
	20%\|██████████████████▍ \| 10/50 [00:05<00:25, 1.59it/s][A
	22%\|████████████████████▏ \| 11/50 [00:06<00:24, 1.58it/s][A
	24%\|██████████████████████ \| 12/50 [00:07<00:24, 1.58it/s][A
	26%\|███████████████████████▉ \| 13/50 [00:07<00:23, 1.58it/s][A
	28%\|█████████████████████████▊ \| 14/50 [00:08<00:22, 1.58it/s][A
	30%\|███████████████████████████▌ \| 15/50 [00:08<00:22, 1.57it/s][A
	32%\|█████████████████████████████▍ \| 16/50 [00:09<00:21, 1.57it/s][A
	34%\|███████████████████████████████▎ \| 17/50 [00:10<00:21, 1.55it/s][A
	36%\|█████████████████████████████████ \| 18/50 [00:10<00:20, 1.56it/s][A
	38%\|██████████████████████████████████▉ \| 19/50 [00:11<00:19, 1.56it/s][A
	40%\|████████████████████████████████████▊ \| 20/50 [00:12<00:19, 1.56it/s][A
	42%\|██████████████████████████████████████▋ \| 21/50 [00:12<00:18, 1.56it/s][A
	44%\|████████████████████████████████████████▍ \| 22/50 [00:13<00:17, 1.57it/s][A
	46%\|██████████████████████████████████████████▎ \| 23/50 [00:14<00:17, 1.57it/s][A
	48%\|████████████████████████████████████████████▏ \| 24/50 [00:14<00:16, 1.57it/s][A
	50%\|██████████████████████████████████████████████ \| 25/50 [00:15<00:16, 1.54it/s][A
	52%\|███████████████████████████████████████████████▊ \| 26/50 [00:16<00:15, 1.55it/s][A
	54%\|█████████████████████████████████████████████████▋ \| 27/50 [00:16<00:14, 1.55it/s][A
	56%\|███████████████████████████████████████████████████▌ \| 28/50 [00:17<00:14, 1.55it/s][A
	58%\|█████████████████████████████████████████████████████▎ \| 29/50 [00:17<00:13, 1.56it/s][A
	60%\|███████████████████████████████████████████████████████▏ \| 30/50 [00:18<00:12, 1.56it/s][A
	62%\|█████████████████████████████████████████████████████████ \| 31/50 [00:19<00:12, 1.57it/s][A
	64%\|██████████████████████████████████████████████████████████▉ \| 32/50 [00:19<00:11, 1.57it/s][A
	66%\|████████████████████████████████████████████████████████████▋ \| 33/50 [00:20<00:11, 1.54it/s][A
	68%\|██████████████████████████████████████████████████████████████▌ \| 34/50 [00:21<00:10, 1.55it/s][A
	70%\|████████████████████████████████████████████████████████████████▍ \| 35/50 [00:21<00:09, 1.56it/s][A
	72%\|██████████████████████████████████████████████████████████████████▏ \| 36/50 [00:22<00:08, 1.56it/s][A
	74%\|████████████████████████████████████████████████████████████████████ \| 37/50 [00:23<00:08, 1.55it/s][A
	76%\|█████████████████████████████████████████████████████████████████████▉ \| 38/50 [00:23<00:07, 1.55it/s][A
	78%\|███████████████████████████████████████████████████████████████████████▊ \| 39/50 [00:24<00:07, 1.56it/s][A
	80%\|█████████████████████████████████████████████████████████████████████████▌ \| 40/50 [00:24<00:06, 1.57it/s][A
	82%\|███████████████████████████████████████████████████████████████████████████▍ \| 41/50 [00:25<00:05, 1.54it/s][A
	84%\|█████████████████████████████████████████████████████████████████████████████▎ \| 42/50 [00:26<00:05, 1.55it/s][A
	86%\|███████████████████████████████████████████████████████████████████████████████ \| 43/50 [00:26<00:04, 1.56it/s][A
	88%\|████████████████████████████████████████████████████████████████████████████████▉ \| 44/50 [00:27<00:03, 1.56it/s][A
	90%\|██████████████████████████████████████████████████████████████████████████████████▊ \| 45/50 [00:28<00:03, 1.56it/s][A
	92%\|████████████████████████████████████████████████████████████████████████████████████▋ \| 46/50 [00:28<00:02, 1.56it/s][A
	94%\|██████████████████████████████████████████████████████████████████████████████████████▍ \| 47/50 [00:29<00:01, 1.56it/s][A
	96%\|████████████████████████████████████████████████████████████████████████████████████████▎ \| 48/50 [00:30<00:01, 1.56it/s][A
	98%\|██████████████████████████████████████████████████████████████████████████████████████████▏ \| 49/50 [00:30<00:00, 1.53it/s][ATraceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-1hkhfopl'
	Traceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-sq1w74l0'
	Traceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-k6gxwff8'
	Traceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-1ne6sntx'

	100%\|████████████████████████████████████████████████████████████████████████████████████████████\| 50/50 [00:31<00:00, 1.51it/s][A
	[A{'eval_loss': 0.8215424418449402, 'eval_runtime': 32.461, 'eval_samples_per_second': 12.322, 'eval_steps_per_second': 1.54, 'eval_ppl': 2.274, 'memory/max_active (GiB)': 42.7, 'memory/max_allocated (GiB)': 42.7, 'memory/device_reserved (GiB)': 60.61, 'epoch': 0.26, 'tokens/train_per_sec_per_gpu': 0.0}
	26%\|████████████████████████▏ \| 15/57 [05:29<12:13, 17.46s/it]
	100%\|████████████████████████████████████████████████████████████████████████████████████████████\| 50/50 [00:31<00:00, 1.51it/s][A
	[A[2026-04-12 22:12:27,880] [INFO] [axolotl.core.trainers.base._save:721] [PID:1885930] Saving model checkpoint to /home/tkwang/scratch/SecSteer-v2/axolotl-outputs/lora/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx0-v2/checkpoint-15
	[2026-04-12 22:12:28,613] [WARNING] [py.warnings._showwarnmsg:112] [PID:1885930] /scratch/tkwang/SecSteer-v2/.venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
	return func(args, *kwargs)

	28%\|█████████████████████████▊ \| 16/57 [05:48<18:54, 27.67s/it] {'loss': 6.0145, 'grad_norm': 0.6384095549583435, 'learning_rate': 3.645967731787313e-05, 'ppl': 409.32113, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.19, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.28}
	28%\|█████████████████████████▊ \| 16/57 [05:48<18:54, 27.67s/it] 30%\|███████████████████████████▍ \| 17/57 [06:05<16:23, 24.58s/it] {'loss': 6.7218, 'grad_norm': 0.7458460330963135, 'learning_rate': 3.5743669612181004e-05, 'ppl': 830.31073, 'memory/max_active (GiB)': 46.13, 'memory/max_allocated (GiB)': 46.13, 'memory/device_reserved (GiB)': 61.19, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.3}
	30%\|███████████████████████████▍ \| 17/57 [06:05<16:23, 24.58s/it] 32%\|█████████████████████████████ \| 18/57 [06:23<14:35, 22.45s/it] {'loss': 6.7857, 'grad_norm': 0.5831446647644043, 'learning_rate': 3.497021496342203e-05, 'ppl': 885.09944, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.19, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.32}
	32%\|█████████████████████████████ \| 18/57 [06:23<14:35, 22.45s/it] 33%\|██████████████████████████████▋ \| 19/57 [06:40<13:15, 20.94s/it] {'loss': 6.87, 'grad_norm': 0.5171515345573425, 'learning_rate': 3.4142135623730954e-05, 'ppl': 962.94857, 'memory/max_active (GiB)': 46.13, 'memory/max_allocated (GiB)': 46.13, 'memory/device_reserved (GiB)': 61.19, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.33}
	33%\|██████████████████████████████▋ \| 19/57 [06:40<13:15, 20.94s/it] 35%\|████████████████████████████████▎ \| 20/57 [06:57<12:15, 19.87s/it] {'loss': 6.3411, 'grad_norm': 0.553388774394989, 'learning_rate': 3.326245316481591e-05, 'ppl': 567.42013, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.19, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.35}
	35%\|████████████████████████████████▎ \| 20/57 [06:57<12:15, 19.87s/it] 37%\|█████████████████████████████████▉ \| 21/57 [07:15<11:29, 19.14s/it] {'loss': 6.6669, 'grad_norm': 0.5413990616798401, 'learning_rate': 3.2334377452570866e-05, 'ppl': 785.95536, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.19, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.37}
	37%\|█████████████████████████████████▉ \| 21/57 [07:15<11:29, 19.14s/it] 39%\|███████████████████████████████████▌ \| 22/57 [07:32<10:53, 18.67s/it] {'loss': 6.5068, 'grad_norm': 0.48956745862960815, 'learning_rate': 3.136129493462312e-05, 'ppl': 669.68001, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.19, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.39}
	39%\|███████████████████████████████████▌ \| 22/57 [07:32<10:53, 18.67s/it] 40%\|█████████████████████████████████████ \| 23/57 [07:50<10:21, 18.28s/it] {'loss': 6.489, 'grad_norm': 0.48526567220687866, 'learning_rate': 3.0346756283553138e-05, 'ppl': 657.86517, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.19, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.4}
	40%\|█████████████████████████████████████ \| 23/57 [07:50<10:21, 18.28s/it] 42%\|██████████████████████████████████████▋ \| 24/57 [08:07<09:53, 17.99s/it] {'loss': 6.6538, 'grad_norm': 0.523944079875946, 'learning_rate': 2.9294463440875375e-05, 'ppl': 775.72649, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.19, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.42}
	42%\|██████████████████████████████████████▋ \| 24/57 [08:07<09:53, 17.99s/it] 44%\|████████████████████████████████████████▎ \| 25/57 [08:25<09:30, 17.81s/it] {'loss': 6.8256, 'grad_norm': 0.4648568630218506, 'learning_rate': 2.820825610905514e-05, 'ppl': 921.12892, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.19, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.44}
	44%\|████████████████████████████████████████▎ \| 25/57 [08:25<09:30, 17.81s/it] 46%\|█████████████████████████████████████████▉ \| 26/57 [08:42<09:08, 17.70s/it] {'loss': 6.9046, 'grad_norm': 0.4684757590293884, 'learning_rate': 2.7092097740850712e-05, 'ppl': 996.84969, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.19, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.46}
	46%\|█████████████████████████████████████████▉ \| 26/57 [08:42<09:08, 17.70s/it] 47%\|███████████████████████████████████████████▌ \| 27/57 [08:59<08:48, 17.63s/it] {'loss': 6.5898, 'grad_norm': 0.4848580062389374, 'learning_rate': 2.595006107710406e-05, 'ppl': 727.63533, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.19, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.47}
	47%\|███████████████████████████████████████████▌ \| 27/57 [08:59<08:48, 17.63s/it] 49%\|█████████████████████████████████████████████▏ \| 28/57 [09:17<08:30, 17.59s/it] {'loss': 6.0481, 'grad_norm': 0.3937620520591736, 'learning_rate': 2.4786313285751158e-05, 'ppl': 423.30798, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.19, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.49}
	49%\|█████████████████████████████████████████████▏ \| 28/57 [09:17<08:30, 17.59s/it] 51%\|██████████████████████████████████████████████▊ \| 29/57 [09:34<08:11, 17.57s/it] {'loss': 6.2209, 'grad_norm': 0.3682299554347992, 'learning_rate': 2.360510075627812e-05, 'ppl': 503.15587, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.19, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.51}
	51%\|██████████████████████████████████████████████▊ \| 29/57 [09:34<08:11, 17.57s/it] 53%\|████████████████████████████████████████████████▍ \| 30/57 [09:52<07:52, 17.50s/it] {'loss': 6.5999, 'grad_norm': 0.47107192873954773, 'learning_rate': 2.2410733605106462e-05, 'ppl': 735.02168, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.19, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.53}
	53%\|████████████████████████████████████████████████▍ \| 30/57 [09:52<07:52, 17.50s/it][2026-04-12 22:16:50,820] [INFO] [axolotl.core.trainers.base.evaluate:400] [PID:1885930] Running evaluation step...

	0%\| \| 0/50 [00:00<?, ?it/s][A
	4%\|███▋ \| 2/50 [00:00<00:15, 3.19it/s][A
	6%\|█████▌ \| 3/50 [00:01<00:21, 2.21it/s][A
	8%\|███████▍ \| 4/50 [00:01<00:23, 1.92it/s][A
	10%\|█████████▎ \| 5/50 [00:02<00:25, 1.79it/s][A
	12%\|███████████▏ \| 6/50 [00:03<00:25, 1.71it/s][A
	14%\|█████████████ \| 7/50 [00:03<00:25, 1.66it/s][A
	16%\|██████████████▉ \| 8/50 [00:04<00:25, 1.64it/s][A
	18%\|████████████████▋ \| 9/50 [00:05<00:25, 1.58it/s][A
	20%\|██████████████████▍ \| 10/50 [00:05<00:25, 1.58it/s][A
	22%\|████████████████████▏ \| 11/50 [00:06<00:24, 1.58it/s][A
	24%\|██████████████████████ \| 12/50 [00:07<00:24, 1.58it/s][A
	26%\|███████████████████████▉ \| 13/50 [00:07<00:23, 1.58it/s][A
	28%\|█████████████████████████▊ \| 14/50 [00:08<00:22, 1.58it/s][A
	30%\|███████████████████████████▌ \| 15/50 [00:08<00:22, 1.57it/s][A
	32%\|█████████████████████████████▍ \| 16/50 [00:09<00:21, 1.57it/s][A
	34%\|███████████████████████████████▎ \| 17/50 [00:10<00:21, 1.55it/s][A
	36%\|█████████████████████████████████ \| 18/50 [00:10<00:20, 1.55it/s][A
	38%\|██████████████████████████████████▉ \| 19/50 [00:11<00:19, 1.56it/s][A
	40%\|████████████████████████████████████▊ \| 20/50 [00:12<00:19, 1.56it/s][A
	42%\|██████████████████████████████████████▋ \| 21/50 [00:12<00:18, 1.56it/s][A
	44%\|████████████████████████████████████████▍ \| 22/50 [00:13<00:17, 1.57it/s][A
	46%\|██████████████████████████████████████████▎ \| 23/50 [00:14<00:17, 1.57it/s][A
	48%\|████████████████████████████████████████████▏ \| 24/50 [00:14<00:16, 1.57it/s][A
	50%\|██████████████████████████████████████████████ \| 25/50 [00:15<00:16, 1.54it/s][A
	52%\|███████████████████████████████████████████████▊ \| 26/50 [00:16<00:15, 1.55it/s][A
	54%\|█████████████████████████████████████████████████▋ \| 27/50 [00:16<00:14, 1.55it/s][A
	56%\|███████████████████████████████████████████████████▌ \| 28/50 [00:17<00:14, 1.55it/s][A
	58%\|█████████████████████████████████████████████████████▎ \| 29/50 [00:17<00:13, 1.56it/s][A
	60%\|███████████████████████████████████████████████████████▏ \| 30/50 [00:18<00:12, 1.56it/s][A
	62%\|█████████████████████████████████████████████████████████ \| 31/50 [00:19<00:12, 1.57it/s][A
	64%\|██████████████████████████████████████████████████████████▉ \| 32/50 [00:19<00:11, 1.57it/s][A
	66%\|████████████████████████████████████████████████████████████▋ \| 33/50 [00:20<00:11, 1.54it/s][A
	68%\|██████████████████████████████████████████████████████████████▌ \| 34/50 [00:21<00:10, 1.55it/s][A
	70%\|████████████████████████████████████████████████████████████████▍ \| 35/50 [00:21<00:09, 1.55it/s][A
	72%\|██████████████████████████████████████████████████████████████████▏ \| 36/50 [00:22<00:08, 1.56it/s][A
	74%\|████████████████████████████████████████████████████████████████████ \| 37/50 [00:23<00:08, 1.55it/s][A
	76%\|█████████████████████████████████████████████████████████████████████▉ \| 38/50 [00:23<00:07, 1.56it/s][A
	78%\|███████████████████████████████████████████████████████████████████████▊ \| 39/50 [00:24<00:07, 1.56it/s][A
	80%\|█████████████████████████████████████████████████████████████████████████▌ \| 40/50 [00:24<00:06, 1.56it/s][A
	82%\|███████████████████████████████████████████████████████████████████████████▍ \| 41/50 [00:25<00:05, 1.54it/s][A
	84%\|█████████████████████████████████████████████████████████████████████████████▎ \| 42/50 [00:26<00:05, 1.55it/s][A
	86%\|███████████████████████████████████████████████████████████████████████████████ \| 43/50 [00:26<00:04, 1.56it/s][A
	88%\|████████████████████████████████████████████████████████████████████████████████▉ \| 44/50 [00:27<00:03, 1.56it/s][A
	90%\|██████████████████████████████████████████████████████████████████████████████████▊ \| 45/50 [00:28<00:03, 1.56it/s][A
	92%\|████████████████████████████████████████████████████████████████████████████████████▋ \| 46/50 [00:28<00:02, 1.56it/s][A
	94%\|██████████████████████████████████████████████████████████████████████████████████████▍ \| 47/50 [00:29<00:01, 1.56it/s][A
	96%\|████████████████████████████████████████████████████████████████████████████████████████▎ \| 48/50 [00:30<00:01, 1.56it/s][A
	98%\|██████████████████████████████████████████████████████████████████████████████████████████▏ \| 49/50 [00:30<00:00, 1.53it/s][ATraceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-_cs3z_tj'
	Traceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-5_xktufy'
	Traceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-tci78w7p'
	Traceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-3j46fqj2'

	100%\|████████████████████████████████████████████████████████████████████████████████████████████\| 50/50 [00:31<00:00, 1.51it/s][A
	[A{'eval_loss': 0.7745330929756165, 'eval_runtime': 32.4547, 'eval_samples_per_second': 12.325, 'eval_steps_per_second': 1.541, 'eval_ppl': 2.16958, 'memory/max_active (GiB)': 42.7, 'memory/max_allocated (GiB)': 42.7, 'memory/device_reserved (GiB)': 61.19, 'epoch': 0.53, 'tokens/train_per_sec_per_gpu': 0.0}
	53%\|████████████████████████████████████████████████▍ \| 30/57 [10:24<07:52, 17.50s/it]
	100%\|████████████████████████████████████████████████████████████████████████████████████████████\| 50/50 [00:31<00:00, 1.51it/s][A
	[A[2026-04-12 22:17:23,303] [INFO] [axolotl.core.trainers.base._save:721] [PID:1885930] Saving model checkpoint to /home/tkwang/scratch/SecSteer-v2/axolotl-outputs/lora/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx0-v2/checkpoint-30
	[2026-04-12 22:17:24,092] [WARNING] [py.warnings._showwarnmsg:112] [PID:1885930] /scratch/tkwang/SecSteer-v2/.venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
	return func(args, *kwargs)

	54%\|██████████████████████████████████████████████████ \| 31/57 [10:43<12:01, 27.74s/it] {'loss': 5.6217, 'grad_norm': 0.4044142961502075, 'learning_rate': 2.1207569948445724e-05, 'ppl': 276.35879, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.19, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.54}
	54%\|██████████████████████████████████████████████████ \| 31/57 [10:43<12:01, 27.74s/it] 56%\|███████████████████████████████████████████████████▋ \| 32/57 [11:01<10:15, 24.64s/it] {'loss': 6.3498, 'grad_norm': 0.4531392455101013, 'learning_rate': 2e-05, 'ppl': 572.37822, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.19, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.56}
	56%\|███████████████████████████████████████████████████▋ \| 32/57 [11:01<10:15, 24.64s/it] 58%\|█████████████████████████████████████████████████████▎ \| 33/57 [11:18<08:58, 22.45s/it] {'loss': 6.2891, 'grad_norm': 0.5013666749000549, 'learning_rate': 1.879243005155428e-05, 'ppl': 538.66831, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.19, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.58}
	58%\|█████████████████████████████████████████████████████▎ \| 33/57 [11:18<08:58, 22.45s/it] 60%\|██████████████████████████████████████████████████████▉ \| 34/57 [11:36<08:01, 20.94s/it] {'loss': 6.1924, 'grad_norm': 0.43094345927238464, 'learning_rate': 1.758926639489354e-05, 'ppl': 489.01834, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.19, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.6}
	60%\|██████████████████████████████████████████████████████▉ \| 34/57 [11:36<08:01, 20.94s/it] 61%\|████████████████████████████████████████████████████████▍ \| 35/57 [11:53<07:18, 19.92s/it] {'loss': 6.0746, 'grad_norm': 0.4792596101760864, 'learning_rate': 1.6394899243721887e-05, 'ppl': 434.6756, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.19, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.61}
	61%\|████████████████████████████████████████████████████████▍ \| 35/57 [11:53<07:18, 19.92s/it] 63%\|██████████████████████████████████████████████████████████ \| 36/57 [12:10<06:41, 19.14s/it] {'loss': 6.0922, 'grad_norm': 0.41381871700286865, 'learning_rate': 1.5213686714248852e-05, 'ppl': 442.39361, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.19, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.63}
	63%\|██████████████████████████████████████████████████████████ \| 36/57 [12:10<06:41, 19.14s/it] 65%\|███████████████████████████████████████████████████████████▋ \| 37/57 [12:28<06:13, 18.66s/it] {'loss': 5.2406, 'grad_norm': 0.3815036416053772, 'learning_rate': 1.4049938922895945e-05, 'ppl': 188.78334, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.19, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.65}
	65%\|███████████████████████████████████████████████████████████▋ \| 37/57 [12:28<06:13, 18.66s/it] 67%\|█████████████████████████████████████████████████████████████▎ \| 38/57 [12:46<05:48, 18.33s/it] {'loss': 6.0361, 'grad_norm': 0.4237765073776245, 'learning_rate': 1.2907902259149287e-05, 'ppl': 418.25864, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.19, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.67}
	67%\|█████████████████████████████████████████████████████████████▎ \| 38/57 [12:46<05:48, 18.33s/it] 68%\|██████████████████████████████████████████████████████████████▉ \| 39/57 [13:03<05:24, 18.04s/it] {'loss': 5.874, 'grad_norm': 0.40918150544166565, 'learning_rate': 1.1791743890944869e-05, 'ppl': 355.66881, 'memory/max_active (GiB)': 46.13, 'memory/max_allocated (GiB)': 46.13, 'memory/device_reserved (GiB)': 61.19, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.68}
	68%\|██████████████████████████████████████████████████████████████▉ \| 39/57 [13:03<05:24, 18.04s/it] 70%\|████████████████████████████████████████████████████████████████▌ \| 40/57 [13:20<05:03, 17.84s/it] {'loss': 6.5058, 'grad_norm': 0.3842076063156128, 'learning_rate': 1.070553655912463e-05, 'ppl': 669.01066, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.19, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.7}
	70%\|████████████████████████████████████████████████████████████████▌ \| 40/57 [13:20<05:03, 17.84s/it] 72%\|██████████████████████████████████████████████████████████████████▏ \| 41/57 [13:38<04:42, 17.68s/it] {'loss': 5.9906, 'grad_norm': 0.46074798703193665, 'learning_rate': 9.653243716446862e-06, 'ppl': 399.65433, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.19, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.72}
	72%\|██████████████████████████████████████████████████████████████████▏ \| 41/57 [13:38<04:42, 17.68s/it] 74%\|███████████████████████████████████████████████████████████████████▊ \| 42/57 [13:55<04:24, 17.61s/it] {'loss': 5.8765, 'grad_norm': 0.36463671922683716, 'learning_rate': 8.638705065376887e-06, 'ppl': 356.5591, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.19, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.74}
	74%\|███████████████████████████████████████████████████████████████████▊ \| 42/57 [13:55<04:24, 17.61s/it] 75%\|█████████████████████████████████████████████████████████████████████▍ \| 43/57 [14:12<04:05, 17.55s/it] {'loss': 6.5206, 'grad_norm': 0.36206111311912537, 'learning_rate': 7.665622547429139e-06, 'ppl': 678.98565, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.19, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.75}
	75%\|█████████████████████████████████████████████████████████████████████▍ \| 43/57 [14:12<04:05, 17.55s/it] 77%\|███████████████████████████████████████████████████████████████████████ \| 44/57 [14:30<03:47, 17.48s/it] {'loss': 6.3755, 'grad_norm': 0.4279758632183075, 'learning_rate': 6.737546835184101e-06, 'ppl': 587.279, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.19, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.77}
	77%\|███████████████████████████████████████████████████████████████████████ \| 44/57 [14:30<03:47, 17.48s/it] 79%\|████████████████████████████████████████████████████████████████████████▋ \| 45/57 [14:47<03:29, 17.47s/it] {'loss': 6.1178, 'grad_norm': 0.3457915484905243, 'learning_rate': 5.857864376269051e-06, 'ppl': 453.86509, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.19, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.79}
	79%\|████████████████████████████████████████████████████████████████████████▋ \| 45/57 [14:47<03:29, 17.47s/it][2026-04-12 22:21:46,224] [INFO] [axolotl.core.trainers.base.evaluate:400] [PID:1885930] Running evaluation step...

	0%\| \| 0/50 [00:00<?, ?it/s][A
	4%\|███▋ \| 2/50 [00:00<00:15, 3.20it/s][A
	6%\|█████▌ \| 3/50 [00:01<00:21, 2.21it/s][A
	8%\|███████▍ \| 4/50 [00:01<00:23, 1.92it/s][A
	10%\|█████████▎ \| 5/50 [00:02<00:25, 1.79it/s][A
	12%\|███████████▏ \| 6/50 [00:03<00:25, 1.71it/s][A
	14%\|█████████████ \| 7/50 [00:03<00:25, 1.66it/s][A
	16%\|██████████████▉ \| 8/50 [00:04<00:25, 1.63it/s][A
	18%\|████████████████▋ \| 9/50 [00:05<00:25, 1.58it/s][A
	20%\|██████████████████▍ \| 10/50 [00:05<00:25, 1.59it/s][A
	22%\|████████████████████▏ \| 11/50 [00:06<00:24, 1.58it/s][A
	24%\|██████████████████████ \| 12/50 [00:07<00:24, 1.58it/s][A
	26%\|███████████████████████▉ \| 13/50 [00:07<00:23, 1.58it/s][A
	28%\|█████████████████████████▊ \| 14/50 [00:08<00:22, 1.58it/s][A
	30%\|███████████████████████████▌ \| 15/50 [00:08<00:22, 1.57it/s][A
	32%\|█████████████████████████████▍ \| 16/50 [00:09<00:21, 1.57it/s][A
	34%\|███████████████████████████████▎ \| 17/50 [00:10<00:21, 1.55it/s][A
	36%\|█████████████████████████████████ \| 18/50 [00:10<00:20, 1.56it/s][A
	38%\|██████████████████████████████████▉ \| 19/50 [00:11<00:19, 1.56it/s][A
	40%\|████████████████████████████████████▊ \| 20/50 [00:12<00:19, 1.56it/s][A
	42%\|██████████████████████████████████████▋ \| 21/50 [00:12<00:18, 1.56it/s][A
	44%\|████████████████████████████████████████▍ \| 22/50 [00:13<00:17, 1.56it/s][A
	46%\|██████████████████████████████████████████▎ \| 23/50 [00:14<00:17, 1.57it/s][A
	48%\|████████████████████████████████████████████▏ \| 24/50 [00:14<00:16, 1.57it/s][A
	50%\|██████████████████████████████████████████████ \| 25/50 [00:15<00:16, 1.54it/s][A
	52%\|███████████████████████████████████████████████▊ \| 26/50 [00:16<00:15, 1.55it/s][A
	54%\|█████████████████████████████████████████████████▋ \| 27/50 [00:16<00:14, 1.55it/s][A
	56%\|███████████████████████████████████████████████████▌ \| 28/50 [00:17<00:14, 1.55it/s][A
	58%\|█████████████████████████████████████████████████████▎ \| 29/50 [00:17<00:13, 1.56it/s][A
	60%\|███████████████████████████████████████████████████████▏ \| 30/50 [00:18<00:12, 1.56it/s][A
	62%\|█████████████████████████████████████████████████████████ \| 31/50 [00:19<00:12, 1.57it/s][A
	64%\|██████████████████████████████████████████████████████████▉ \| 32/50 [00:19<00:11, 1.57it/s][A
	66%\|████████████████████████████████████████████████████████████▋ \| 33/50 [00:20<00:11, 1.54it/s][A
	68%\|██████████████████████████████████████████████████████████████▌ \| 34/50 [00:21<00:10, 1.55it/s][A
	70%\|████████████████████████████████████████████████████████████████▍ \| 35/50 [00:21<00:09, 1.55it/s][A
	72%\|██████████████████████████████████████████████████████████████████▏ \| 36/50 [00:22<00:08, 1.56it/s][A
	74%\|████████████████████████████████████████████████████████████████████ \| 37/50 [00:23<00:08, 1.55it/s][A
	76%\|█████████████████████████████████████████████████████████████████████▉ \| 38/50 [00:23<00:07, 1.55it/s][A
	78%\|███████████████████████████████████████████████████████████████████████▊ \| 39/50 [00:24<00:07, 1.56it/s][A
	80%\|█████████████████████████████████████████████████████████████████████████▌ \| 40/50 [00:24<00:06, 1.56it/s][A
	82%\|███████████████████████████████████████████████████████████████████████████▍ \| 41/50 [00:25<00:05, 1.54it/s][A
	84%\|█████████████████████████████████████████████████████████████████████████████▎ \| 42/50 [00:26<00:05, 1.55it/s][A
	86%\|███████████████████████████████████████████████████████████████████████████████ \| 43/50 [00:26<00:04, 1.56it/s][A
	88%\|████████████████████████████████████████████████████████████████████████████████▉ \| 44/50 [00:27<00:03, 1.56it/s][A
	90%\|██████████████████████████████████████████████████████████████████████████████████▊ \| 45/50 [00:28<00:03, 1.56it/s][A
	92%\|████████████████████████████████████████████████████████████████████████████████████▋ \| 46/50 [00:28<00:02, 1.56it/s][A
	94%\|██████████████████████████████████████████████████████████████████████████████████████▍ \| 47/50 [00:29<00:01, 1.56it/s][A
	96%\|████████████████████████████████████████████████████████████████████████████████████████▎ \| 48/50 [00:30<00:01, 1.56it/s][A
	98%\|██████████████████████████████████████████████████████████████████████████████████████████▏ \| 49/50 [00:30<00:00, 1.53it/s][ATraceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-tmm5kofe'
	Traceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-qdphuqjr'
	Traceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-ggl4tm3l'
	Traceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-r82dkx52'

	100%\|████████████████████████████████████████████████████████████████████████████████████████████\| 50/50 [00:31<00:00, 1.48it/s][A
	[A{'eval_loss': 0.7658505439758301, 'eval_runtime': 32.4836, 'eval_samples_per_second': 12.314, 'eval_steps_per_second': 1.539, 'eval_ppl': 2.15082, 'memory/max_active (GiB)': 42.7, 'memory/max_allocated (GiB)': 42.7, 'memory/device_reserved (GiB)': 61.19, 'epoch': 0.79, 'tokens/train_per_sec_per_gpu': 0.0}
	79%\|████████████████████████████████████████████████████████████████████████▋ \| 45/57 [15:20<03:29, 17.47s/it]
	100%\|████████████████████████████████████████████████████████████████████████████████████████████\| 50/50 [00:31<00:00, 1.48it/s][A
	[A[2026-04-12 22:22:18,794] [INFO] [axolotl.core.trainers.base._save:721] [PID:1885930] Saving model checkpoint to /home/tkwang/scratch/SecSteer-v2/axolotl-outputs/lora/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx0-v2/checkpoint-45
	[2026-04-12 22:22:19,591] [WARNING] [py.warnings._showwarnmsg:112] [PID:1885930] /scratch/tkwang/SecSteer-v2/.venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
	return func(args, *kwargs)

	81%\|██████████████████████████████████████████████████████████████████████████▏ \| 46/57 [15:39<05:04, 27.72s/it] {'loss': 6.7799, 'grad_norm': 0.4133686423301697, 'learning_rate': 5.029785036577976e-06, 'ppl': 879.98072, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.18, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.81}
	81%\|██████████████████████████████████████████████████████████████████████████▏ \| 46/57 [15:39<05:04, 27.72s/it] 82%\|███████████████████████████████████████████████████████████████████████████▊ \| 47/57 [15:56<04:06, 24.66s/it] {'loss': 5.4688, 'grad_norm': 0.3395274877548218, 'learning_rate': 4.256330387818999e-06, 'ppl': 237.17541, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.18, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.82}
	82%\|███████████████████████████████████████████████████████████████████████████▊ \| 47/57 [15:56<04:06, 24.66s/it] 84%\|█████████████████████████████████████████████████████████████████████████████▍ \| 48/57 [16:14<03:22, 22.49s/it] {'loss': 6.0719, 'grad_norm': 0.3674202561378479, 'learning_rate': 3.5403226821268734e-06, 'ppl': 433.50356, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.18, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.84}
	84%\|█████████████████████████████████████████████████████████████████████████████▍ \| 48/57 [16:14<03:22, 22.49s/it] 86%\|███████████████████████████████████████████████████████████████████████████████ \| 49/57 [16:31<02:47, 20.94s/it] {'loss': 5.9251, 'grad_norm': 0.3795262277126312, 'learning_rate': 2.8843745539710523e-06, 'ppl': 374.31587, 'memory/max_active (GiB)': 46.13, 'memory/max_allocated (GiB)': 46.13, 'memory/device_reserved (GiB)': 61.18, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.86}
	86%\|███████████████████████████████████████████████████████████████████████████████ \| 49/57 [16:31<02:47, 20.94s/it] 88%\|████████████████████████████████████████████████████████████████████████████████▋ \| 50/57 [16:49<02:19, 19.89s/it] {'loss': 5.9377, 'grad_norm': 0.4174330532550812, 'learning_rate': 2.2908794869358044e-06, 'ppl': 379.06208, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.19, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.88}
	88%\|████████████████████████████████████████████████████████████████████████████████▋ \| 50/57 [16:49<02:19, 19.89s/it] 89%\|██████████████████████████████████████████████████████████████████████████████████▎ \| 51/57 [17:06<01:54, 19.14s/it] {'loss': 6.0881, 'grad_norm': 0.37792524695396423, 'learning_rate': 1.7620030801581988e-06, 'ppl': 440.58351, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.19, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.89}
	89%\|██████████████████████████████████████████████████████████████████████████████████▎ \| 51/57 [17:06<01:54, 19.14s/it] 91%\|███████████████████████████████████████████████████████████████████████████████████▉ \| 52/57 [17:23<01:33, 18.60s/it] {'loss': 6.8544, 'grad_norm': 0.41102176904678345, 'learning_rate': 1.2996751462917057e-06, 'ppl': 948.04313, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.19, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.91}
	91%\|███████████████████████████████████████████████████████████████████████████████████▉ \| 52/57 [17:23<01:33, 18.60s/it] 93%\|█████████████████████████████████████████████████████████████████████████████████████▌ \| 53/57 [17:41<01:13, 18.26s/it] {'loss': 5.8281, 'grad_norm': 0.39307862520217896, 'learning_rate': 9.055826698290881e-07, 'ppl': 339.71261, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.19, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.93}
	93%\|█████████████████████████████████████████████████████████████████████████████████████▌ \| 53/57 [17:41<01:13, 18.26s/it] 95%\|███████████████████████████████████████████████████████████████████████████████████████▏ \| 54/57 [17:58<00:54, 18.01s/it] {'loss': 6.7507, 'grad_norm': 0.40890562534332275, 'learning_rate': 5.811636514789598e-07, 'ppl': 854.65681, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.19, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.95}
	95%\|███████████████████████████████████████████████████████████████████████████████████████▏ \| 54/57 [17:58<00:54, 18.01s/it] 96%\|████████████████████████████████████████████████████████████████████████████████████████▊ \| 55/57 [18:16<00:35, 17.82s/it] {'loss': 6.3531, 'grad_norm': 0.42297133803367615, 'learning_rate': 3.2760186105712964e-07, 'ppl': 574.27019, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.19, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.96}
	96%\|████████████████████████████████████████████████████████████████████████████████████████▊ \| 55/57 [18:16<00:35, 17.82s/it] 98%\|██████████████████████████████████████████████████████████████████████████████████████████▍ \| 56/57 [18:33<00:17, 17.74s/it] {'loss': 5.604, 'grad_norm': 0.37047332525253296, 'learning_rate': 1.4582251803892055e-07, 'ppl': 271.51028, 'memory/max_active (GiB)': 46.14, 'memory/max_allocated (GiB)': 46.14, 'memory/device_reserved (GiB)': 61.19, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 0.98}
	98%\|██████████████████████████████████████████████████████████████████████████████████████████▍ \| 56/57 [18:33<00:17, 17.74s/it]Traceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-cc5dnmmu'
	Traceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-49010n8h'
	Traceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-6ze5a2tl'
	Traceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-alzp1ojx'
	100%\|████████████████████████████████████████████████████████████████████████████████████████████\| 57/57 [18:51<00:00, 17.68s/it] {'loss': 5.6777, 'grad_norm': 0.3775448203086853, 'learning_rate': 3.648891553365008e-08, 'ppl': 292.27642, 'memory/max_active (GiB)': 46.13, 'memory/max_allocated (GiB)': 46.13, 'memory/device_reserved (GiB)': 61.19, 'tokens/train_per_sec_per_gpu': 0.0, 'tokens/total': 0, 'tokens/trainable': 0, 'epoch': 1.0}
	100%\|████████████████████████████████████████████████████████████████████████████████████████████\| 57/57 [18:51<00:00, 17.68s/it][2026-04-12 22:25:49,699] [INFO] [axolotl.core.trainers.base._save:721] [PID:1885930] Saving model checkpoint to /home/tkwang/scratch/SecSteer-v2/axolotl-outputs/lora/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx0-v2/checkpoint-57
	[2026-04-12 22:25:50,813] [WARNING] [py.warnings._showwarnmsg:112] [PID:1885930] /scratch/tkwang/SecSteer-v2/.venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
	return func(args, *kwargs)

	{'train_runtime': 1136.5199, 'train_samples_per_second': 3.21, 'train_steps_per_second': 0.05, 'train_loss': 6.416730939296254, 'memory/max_active (GiB)': 15.01, 'memory/max_allocated (GiB)': 15.01, 'memory/device_reserved (GiB)': 61.19, 'epoch': 1.0, 'tokens/train_per_sec_per_gpu': 0.0}
	100%\|████████████████████████████████████████████████████████████████████████████████████████████\| 57/57 [18:53<00:00, 17.68s/it] 100%\|████████████████████████████████████████████████████████████████████████████████████████████\| 57/57 [18:53<00:00, 19.88s/it]
	[2026-04-12 22:25:51,750] [INFO] [axolotl.train.save_trained_model:233] [PID:1885930] Training completed! Saving trained model to /home/tkwang/scratch/SecSteer-v2/axolotl-outputs/lora/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx0-v2.
	[2026-04-12 22:25:52,374] [INFO] [axolotl.train.save_trained_model:351] [PID:1885930] Model successfully saved to /home/tkwang/scratch/SecSteer-v2/axolotl-outputs/lora/Qwen2.5-Coder-7B-sft-plus-alpha-1-token-diff-ctx0-v2