Instructions to use felixwangg/Qwen2.5-Coder-7B-func-stage1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use felixwangg/Qwen2.5-Coder-7B-func-stage1 with PEFT:

from peft import PeftModel
from transformers import AutoModelForCausalLM

base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-7B-Instruct")
model = PeftModel.from_pretrained(base_model, "felixwangg/Qwen2.5-Coder-7B-func-stage1")

Transformers

How to use felixwangg/Qwen2.5-Coder-7B-func-stage1 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="felixwangg/Qwen2.5-Coder-7B-func-stage1")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("felixwangg/Qwen2.5-Coder-7B-func-stage1")
model = AutoModelForCausalLM.from_pretrained("felixwangg/Qwen2.5-Coder-7B-func-stage1")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use felixwangg/Qwen2.5-Coder-7B-func-stage1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "felixwangg/Qwen2.5-Coder-7B-func-stage1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "felixwangg/Qwen2.5-Coder-7B-func-stage1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/felixwangg/Qwen2.5-Coder-7B-func-stage1

SGLang

How to use felixwangg/Qwen2.5-Coder-7B-func-stage1 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "felixwangg/Qwen2.5-Coder-7B-func-stage1" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "felixwangg/Qwen2.5-Coder-7B-func-stage1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "felixwangg/Qwen2.5-Coder-7B-func-stage1" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "felixwangg/Qwen2.5-Coder-7B-func-stage1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use felixwangg/Qwen2.5-Coder-7B-func-stage1 with Docker Model Runner:
```
docker model run hf.co/felixwangg/Qwen2.5-Coder-7B-func-stage1
```

Qwen2.5-Coder-7B-func-stage1 / debug.log

felixwangg

Upload folder using huggingface_hub

92449a3 verified about 1 month ago

raw

history blame contribute delete

90.9 kB

	[2026-04-21 00:44:50,108] [DEBUG] [axolotl.utils.config.log_gpu_memory_usage:127] [PID:4071480] baseline 0.000GB ()
	[2026-04-21 00:44:50,108] [INFO] [axolotl.cli.config.load_cfg:341] [PID:4071480] config:
	{
	"activation_offloading": false,
	"adapter": "lora",
	"axolotl_config_path": "./axolotl_configs/Qwen2.5-coder-7b-instruct/func_stage1_sec_stage2/lora-func_stage_1.yaml",
	"base_model": "Qwen/Qwen2.5-Coder-7B-Instruct",
	"base_model_config": "Qwen/Qwen2.5-Coder-7B-Instruct",
	"batch_size": 64,
	"bf16": true,
	"capabilities": {
	"bf16": true,
	"compute_capability": "sm_90",
	"fp8": true,
	"n_gpu": 2,
	"n_node": 1,
	"tf32": true
	},
	"context_parallel_size": 1,
	"dataloader_num_workers": 2,
	"dataloader_pin_memory": true,
	"dataloader_prefetch_factor": 256,
	"dataset_num_proc": 112,
	"dataset_prepared_path": "/home/tkwang/scratch/SecSteer-v2/axolotl-datasets/lora/Qwen2.5-Coder-7B/func-stage1",
	"datasets": [
	{
	"chat_template": "tokenizer_default",
	"message_property_mappings": {
	"content": "content",
	"role": "role"
	},
	"path": "felixwangg/codenet-c-cpp-stage1",
	"split": "train",
	"trust_remote_code": false,
	"type": "chat_template"
	}
	],
	"ddp": true,
	"device": "cuda:0",
	"device_map": {
	"": 0
	},
	"dion_rank_fraction": 1.0,
	"dion_rank_multiple_of": 1,
	"eaft_alpha": 1.0,
	"eaft_k": 20,
	"early_stopping_patience": 1000,
	"env_capabilities": {
	"torch_version": "2.11.0"
	},
	"eval_batch_size": 4,
	"eval_causal_lm_metrics": [
	"sacrebleu",
	"comet",
	"ter",
	"chrf"
	],
	"eval_max_new_tokens": 128,
	"eval_sample_packing": false,
	"eval_steps": 5,
	"eval_table_size": 0,
	"experimental_skip_move_to_device": true,
	"flash_attention": true,
	"fp16": false,
	"generate_samples": false,
	"generation_do_sample": true,
	"generation_max_new_tokens": 50,
	"generation_prompt_ratio": 0.5,
	"generation_temperature": 0.7,
	"gradient_accumulation_steps": 8,
	"gradient_checkpointing": true,
	"gradient_checkpointing_kwargs": {
	"use_reentrant": true
	},
	"include_tkps": true,
	"is_falcon_derived_model": false,
	"is_llama_derived_model": false,
	"is_mistral_derived_model": false,
	"layer_offloading": false,
	"learning_rate": 4e-05,
	"lisa_layers_attribute": "model.layers",
	"load_best_model_at_end": true,
	"load_in_4bit": false,
	"load_in_8bit": false,
	"local_rank": 0,
	"logging_steps": 1,
	"lora_alpha": 16,
	"lora_dropout": 0.05,
	"lora_embedding_kernel": true,
	"lora_mlp_kernel": true,
	"lora_o_kernel": true,
	"lora_qkv_kernel": true,
	"lora_r": 16,
	"lora_target_linear": true,
	"loraplus_lr_embedding": 1e-06,
	"lr_scheduler": "cosine",
	"mean_resizing_embeddings": false,
	"merge_lora": true,
	"merge_method": "memory_efficient",
	"micro_batch_size": 4,
	"model_config_type": "qwen2",
	"num_epochs": 1.0,
	"num_generation_samples": 3,
	"optimizer": "adamw_torch",
	"otel_metrics_host": "localhost",
	"otel_metrics_port": 8000,
	"output_dir": "/home/tkwang/scratch/SecSteer-v2/axolotl-outputs/lora/Qwen2.5-Coder-7B-func-stage1",
	"pad_to_sequence_len": true,
	"pretrain_multipack_attn": true,
	"profiler_steps_start": 0,
	"qlora_sharded_model_loading": false,
	"quantize_moe_experts": false,
	"ray_num_workers": 1,
	"resources_per_worker": {
	"GPU": 1
	},
	"sample_packing": false,
	"sample_packing_bin_size": 200,
	"sample_packing_group_size": 100000,
	"save_only_model": false,
	"save_safetensors": true,
	"save_steps": 5,
	"save_total_limit": 1000,
	"sequence_len": 4096,
	"shuffle_before_merging_datasets": false,
	"shuffle_merged_datasets": true,
	"skip_prepare_dataset": false,
	"streaming_multipack_buffer_size": 10000,
	"strict": false,
	"tensor_parallel_size": 1,
	"test_datasets": [
	{
	"chat_template": "tokenizer_default",
	"message_property_mappings": {
	"content": "content",
	"role": "role"
	},
	"path": "felixwangg/codenet-c-cpp-stage1",
	"split": "validation",
	"trust_remote_code": false,
	"type": "chat_template"
	}
	],
	"tf32": false,
	"tiled_mlp_use_original_mlp": true,
	"tokenizer_config": "Qwen/Qwen2.5-Coder-7B-Instruct",
	"tokenizer_save_jinja_files": true,
	"tokenizer_type": "AutoTokenizer",
	"torch_dtype": "torch.bfloat16",
	"train_on_inputs": false,
	"trl": {
	"async_prefetch": false,
	"log_completions": false,
	"mask_truncated_completions": false,
	"ref_model_mixup_alpha": 0.9,
	"ref_model_sync_steps": 64,
	"replay_buffer_size": 0,
	"replay_recompute_logps": true,
	"reroll_max_groups": 1,
	"reroll_start_fraction": 1.0,
	"reward_num_workers": 1,
	"scale_rewards": true,
	"skip_zero_advantage_batches": true,
	"sync_ref_model": false,
	"use_data_producer": false,
	"use_vllm": false,
	"vllm_lora_sync": false,
	"vllm_server_host": "0.0.0.0",
	"vllm_server_port": 8000
	},
	"type_of_model": "Qwen2ForCausalLM",
	"use_otel_metrics": false,
	"use_ray": false,
	"use_wandb": true,
	"val_set_size": 0.0,
	"vllm": {
	"device": "auto",
	"dtype": "auto",
	"gpu_memory_utilization": 0.9,
	"host": "0.0.0.0",
	"port": 8000
	},
	"wandb_entity": "wtkuan",
	"wandb_log_model": "false",
	"wandb_name": "Qwen2.5-Coder-7B-func-stage1",
	"wandb_project": "sft-primevul-sweep-ctx-0",
	"wandb_watch": "false",
	"warmup_ratio": 0.1,
	"weight_decay": 0.02,
	"world_size": 2
	}
	[2026-04-21 00:44:50,754] [DEBUG] [axolotl.loaders.tokenizer.load_tokenizer:307] [PID:4071480] EOS: 151645 / <\|im_end\|>
	[2026-04-21 00:44:50,754] [DEBUG] [axolotl.loaders.tokenizer.load_tokenizer:308] [PID:4071480] BOS: None / None
	[2026-04-21 00:44:50,754] [DEBUG] [axolotl.loaders.tokenizer.load_tokenizer:309] [PID:4071480] PAD: 151643 / <\|endoftext\|>
	[2026-04-21 00:44:50,754] [DEBUG] [axolotl.loaders.tokenizer.load_tokenizer:310] [PID:4071480] UNK: None / None
	[2026-04-21 00:44:50,861] [INFO] [axolotl.utils.data.shared.load_preprocessed_dataset:475] [PID:4071480] Loading prepared dataset from disk at /home/tkwang/scratch/SecSteer-v2/axolotl-datasets/lora/Qwen2.5-Coder-7B/func-stage1/96a3e20eec657a543cd77229b2f68312...
	[2026-04-21 00:44:50,870] [INFO] [axolotl.utils.data.shared.load_preprocessed_dataset:475] [PID:4071480] Loading prepared dataset from disk at /home/tkwang/scratch/SecSteer-v2/axolotl-datasets/lora/Qwen2.5-Coder-7B/func-stage1/2f9b6666b4ccc6f854316e096ff5dcca...
	[2026-04-21 00:44:50,922] [DEBUG] [axolotl.utils.trainer.calculate_total_num_steps:420] [PID:4071480] total_num_tokens: 1_094_060
	[2026-04-21 00:44:50,927] [DEBUG] [axolotl.utils.trainer.calculate_total_num_steps:438] [PID:4071480] `total_supervised_tokens: 536_809`
	[2026-04-21 00:44:50,927] [DEBUG] [axolotl.utils.trainer.calculate_total_num_steps:521] [PID:4071480] total_num_steps: 15
	[2026-04-21 00:44:50,927] [INFO] [axolotl.utils.data.sft._prepare_standard_dataset:121] [PID:4071480] Maximum number of steps set at 15
	[2026-04-21 00:44:50,952] [DEBUG] [axolotl.train.setup_model_and_tokenizer:70] [PID:4071480] loading tokenizer... Qwen/Qwen2.5-Coder-7B-Instruct
	[2026-04-21 00:44:51,508] [DEBUG] [axolotl.loaders.tokenizer.load_tokenizer:307] [PID:4071480] EOS: 151645 / <\|im_end\|>
	[2026-04-21 00:44:51,508] [DEBUG] [axolotl.loaders.tokenizer.load_tokenizer:308] [PID:4071480] BOS: None / None
	[2026-04-21 00:44:51,508] [DEBUG] [axolotl.loaders.tokenizer.load_tokenizer:309] [PID:4071480] PAD: 151643 / <\|endoftext\|>
	[2026-04-21 00:44:51,508] [DEBUG] [axolotl.loaders.tokenizer.load_tokenizer:310] [PID:4071480] UNK: None / None
	[2026-04-21 00:44:51,508] [DEBUG] [axolotl.train.setup_model_and_tokenizer:81] [PID:4071480] Loading model
	[2026-04-21 00:44:51,579] [DEBUG] [axolotl.monkeypatch.torchao_optim.patch_torchao_optim_state_8bit:75] [PID:4071480] Patched OptimState8bit for torch.compile compatibility
	[2026-04-21 00:44:51,579] [DEBUG] [axolotl.monkeypatch.torchao_optim.patch_torchao_optim_state_8bit:122] [PID:4071480] Patched OptimState4bit for torch.compile compatibility
	[2026-04-21 00:44:51,579] [DEBUG] [axolotl.monkeypatch.torchao_optim.patch_torchao_optim_state_8bit:154] [PID:4071480] Patched OptimStateFp8 for torch.compile compatibility
	[2026-04-21 00:44:51,583] [DEBUG] [axolotl.monkeypatch.transformers.trainer_loss_calc.patch_evaluation_loop:94] [PID:4071480] Patched Trainer.evaluation_loop with nanmean loss calculation
	[2026-04-21 00:44:51,584] [DEBUG] [axolotl.monkeypatch.transformers.trainer_loss_calc.patch_maybe_log_save_evaluate:148] [PID:4071480] Patched Trainer._maybe_log_save_evaluate with nanmean loss calculation
	[2026-04-21 00:44:51,587] [INFO] [axolotl.monkeypatch.attention.flash_attn_4.patch_flash_attn_4:52] [PID:4071480] Flash Attention 4 is available for your GPU and offers faster training speeds. To enable: pip install flash-attn-4
	[2026-04-21 00:44:51,587] [WARNING] [axolotl.loaders.patch_manager._apply_self_attention_lora_patch:436] [PID:4071480] Cannot patch self-attention - requires no dropout
	Downloading (incomplete total...): 0.00B [00:00, ?B/s]
	Fetching 22 files: 0%\| \| 0/22 [00:00<?, ?it/s][A Downloading (incomplete total...): 0.00B [00:00, ?B/s]
	Fetching 22 files: 0%\| \| 0/22 [00:00<?, ?it/s] Fetching 22 files: 100%\|█████████████████████████████████████████████████████████████████████████████████\| 22/22 [00:00<00:00, 2050.05it/s][A
	Download complete: : 0.00B [00:00, ?B/s] Fetching 22 files: 100%\|█████████████████████████████████████████████████████████████████████████████████\| 22/22 [00:00<00:00, 8645.62it/s]
	Download complete: : 0.00B [00:00, ?B/s] Download complete: : 0.00B [00:00, ?B/s] Download complete: : 0.00B [00:00, ?B/s]
	Loading weights: 0%\| \| 0/339 [00:00<?, ?it/s]
	Loading weights: 0%\| \| 0/339 [00:00<?, ?it/s] Loading weights: 0%\|▎ \| 1/339 [00:00<00:58, 5.81it/s] Loading weights: 0%\|▎ \| 1/339 [00:00<00:57, 5.93it/s] Loading weights: 1%\|▌ \| 2/339 [00:00<01:01, 5.50it/s] Loading weights: 1%\|▌ \| 2/339 [00:00<01:01, 5.46it/s] Loading weights: 4%\|██▉ \| 12/339 [00:00<00:10, 32.70it/s] Loading weights: 2%\|█▌ \| 6/339 [00:00<00:31, 10.50it/s] Loading weights: 6%\|█████▏ \| 21/339 [00:00<00:08, 37.85it/s] Loading weights: 16%\|█████████████▍ \| 54/339 [00:00<00:03, 92.98it/s] Loading weights: 29%\|████████████████████████▏ \| 100/339 [00:00<00:01, 162.46it/s] Loading weights: 36%\|█████████████████████████████▎ \| 121/339 [00:01<00:01, 136.09it/s] Loading weights: 42%\|██████████████████████████████████▊ \| 144/339 [00:01<00:01, 122.25it/s] Loading weights: 17%\|██████████████ \| 57/339 [00:01<00:07, 38.38it/s] Loading weights: 56%\|█████████████████████████████████████████████▋ \| 189/339 [00:01<00:01, 140.58it/s] Loading weights: 66%\|██████████████████████████████████████████████████████▏ \| 224/339 [00:01<00:00, 166.62it/s] Loading weights: 73%\|████████████████████████████████████████████████████████████▏ \| 249/339 [00:02<00:00, 144.34it/s] Loading weights: 67%\|███████████████████████████████████████████████████████▏ \| 228/339 [00:02<00:00, 149.27it/s] Loading weights: 88%\|███████████████████████████████████████████████████████████████████████▊ \| 297/339 [00:02<00:00, 191.79it/s] Loading weights: 88%\|████████████████████████████████████████████████████████████████████████▌ \| 300/339 [00:02<00:00, 185.72it/s] Loading weights: 94%\|█████████████████████████████████████████████████████████████████████████████▍ \| 320/339 [00:02<00:00, 186.43it/s] Loading weights: 100%\|██████████████████████████████████████████████████████████████████████████████████\| 339/339 [00:02<00:00, 145.33it/s] Loading weights: 100%\|██████████████████████████████████████████████████████████████████████████████████\| 339/339 [00:02<00:00, 145.12it/s]
	[2026-04-21 00:44:55,034] [INFO] [axolotl.loaders.model._configure_embedding_dtypes:361] [PID:4071480] Converting modules to torch.bfloat16
	[2026-04-21 00:44:55,035] [DEBUG] [axolotl.loaders.model.log_gpu_memory_usage:127] [PID:4071480] Memory usage after model load 17.234GB (+17.234GB allocated, +18.250GB reserved)
	[2026-04-21 00:44:55,036] [INFO] [axolotl.loaders.adapter.load_lora:81] [PID:4071480] found linear modules: ['down_proj', 'gate_proj', 'k_proj', 'o_proj', 'q_proj', 'up_proj', 'v_proj']
	trainable params: 40,370,176 \|\| all params: 7,655,986,688 \|\| trainable%: 0.5273
	[2026-04-21 00:44:55,535] [DEBUG] [axolotl.loaders.model.log_gpu_memory_usage:127] [PID:4071480] after adapters 14.337GB (+14.337GB allocated, +18.328GB reserved)
	[2026-04-21 00:44:58,236] [INFO] [axolotl.monkeypatch.lora_kernels.apply_lora_kernel_patches:436] [PID:4071480] LoRA kernels: dropout=0.05 enabled
	[2026-04-21 00:44:58,813] [INFO] [axolotl.train.save_initial_configs:417] [PID:4071480] Pre-saving adapter config to /home/tkwang/scratch/SecSteer-v2/axolotl-outputs/lora/Qwen2.5-Coder-7B-func-stage1...
	[2026-04-21 00:44:58,827] [INFO] [axolotl.train.save_initial_configs:421] [PID:4071480] Pre-saving tokenizer to /home/tkwang/scratch/SecSteer-v2/axolotl-outputs/lora/Qwen2.5-Coder-7B-func-stage1...
	[2026-04-21 00:44:58,988] [INFO] [axolotl.train.save_initial_configs:426] [PID:4071480] Pre-saving model config to /home/tkwang/scratch/SecSteer-v2/axolotl-outputs/lora/Qwen2.5-Coder-7B-func-stage1...
	[2026-04-21 00:44:59,003] [INFO] [axolotl.train.execute_training:222] [PID:4071480] Starting trainer...

	[34m[1mwandb[0m: [wandb.login()] Loaded credentials for https://api.wandb.ai from /home/tkwang/.netrc.
	[34m[1mwandb[0m: Currently logged in as: [33mwtkuan[0m to [32mhttps://api.wandb.ai[0m. Use [1m`wandb login --relogin`[0m to force relogin
	[34m[1mwandb[0m: Using an existing wandb-core service via WANDB_SERVICE.
	[34m[1mwandb[0m: [38;5;178m⢿[0m Waiting for wandb.init()...
	[Am[2K [34m[1mwandb[0m: [38;5;178m⣻[0m Waiting for wandb.init()...
	[Am[2K [34m[1mwandb[0m: [38;5;178m⣽[0m Waiting for wandb.init()...
	[Am[2K [34m[1mwandb[0m: [38;5;178m⣾[0m Waiting for wandb.init()...
	[Am[2K [34m[1mwandb[0m: [38;5;178m⣷[0m setting up run zy341u4l (0.4s)
	[Am[2K [34m[1mwandb[0m: [38;5;178m⣯[0m setting up run zy341u4l (0.4s)
	[Am[2K [34m[1mwandb[0m: [38;5;178m⣟[0m setting up run zy341u4l (0.4s)
	[Am[2K [34m[1mwandb[0m: [38;5;178m⡿[0m setting up run zy341u4l (0.4s)
	[Am[2K [34m[1mwandb[0m: Tracking run with wandb version 0.26.0
	[34m[1mwandb[0m: Run data is saved locally in [35m[1m/scratch/tkwang/SecSteer-v2/wandb/run-20260421_004500-zy341u4l[0m
	[34m[1mwandb[0m: Run [1m`wandb offline`[0m to turn off syncing.
	[34m[1mwandb[0m: Syncing run [33mQwen2.5-Coder-7B-func-stage1[0m
	[34m[1mwandb[0m: ⭐️ View project at [34m[4mhttps://wandb.ai/wtkuan/sft-primevul-sweep-ctx-0[0m
	[34m[1mwandb[0m: 🚀 View run at [34m[4mhttps://wandb.ai/wtkuan/sft-primevul-sweep-ctx-0/runs/zy341u4l[0m
	[34m[1mwandb[0m: [33mWARNING[0m Saving files without folders. If you want to preserve subdirectories pass base_path to wandb.save, i.e. wandb.save("/mnt/folder/file.h5", base_path="/mnt")
	[34m[1mwandb[0m: [33mWARNING[0m Symlinked 1 file into the W&B run directory; call wandb.save again to sync new files.
	[2026-04-21 00:45:03,417] [INFO] [axolotl.utils.callbacks.on_train_begin:757] [PID:4071480] The Axolotl config has been saved to the WandB run under files.
	0%\| \| 0/15 [00:00<?, ?it/s][2026-04-21 00:45:03,419] [INFO] [axolotl.core.trainers.base.evaluate:410] [PID:4071480] Running evaluation step...

	0%\| \| 0/13 [00:00<?, ?it/s][A
	15%\|███████████████▊ \| 2/13 [00:00<00:02, 4.02it/s][A
	23%\|███████████████████████▊ \| 3/13 [00:01<00:03, 2.65it/s][A
	31%\|███████████████████████████████▋ \| 4/13 [00:01<00:04, 2.21it/s][A
	38%\|███████████████████████████████████████▌ \| 5/13 [00:02<00:03, 2.05it/s][A
	46%\|███████████████████████████████████████████████▌ \| 6/13 [00:02<00:03, 1.96it/s][A
	54%\|███████████████████████████████████████████████████████▍ \| 7/13 [00:03<00:03, 1.90it/s][A
	62%\|███████████████████████████████████████████████████████████████▍ \| 8/13 [00:03<00:02, 1.86it/s][A
	69%\|███████████████████████████████████████████████████████████████████████▎ \| 9/13 [00:04<00:02, 1.80it/s][A
	77%\|██████████████████████████████████████████████████████████████████████████████▍ \| 10/13 [00:05<00:01, 1.79it/s][A
	85%\|██████████████████████████████████████████████████████████████████████████████████████▎ \| 11/13 [00:05<00:01, 1.78it/s][A
	92%\|██████████████████████████████████████████████████████████████████████████████████████████████▏ \| 12/13 [00:06<00:00, 1.77it/s][ATraceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-mwp73n5z'
	Traceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-0z54svfe'
	Traceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-pd6k1714'
	Traceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-kh50wmwv'

	100%\|██████████████████████████████████████████████████████████████████████████████████████████████████████\| 13/13 [00:06<00:00, 1.75it/s][A
	[A{'eval_loss': '0.6554', 'eval_runtime': '8.209', 'eval_samples_per_second': '12.18', 'eval_steps_per_second': '1.584', 'eval_ppl': '1.926', 'memory/max_active (GiB)': '37.85', 'memory/max_allocated (GiB)': '37.85', 'memory/device_reserved (GiB)': '41.82', 'epoch': 0}
	0%\| \| 0/15 [00:08<?, ?it/s]
	100%\|██████████████████████████████████████████████████████████████████████████████████████████████████████\| 13/13 [00:06<00:00, 1.75it/s][A
	[A 7%\|██████▊ \| 1/15 [00:23<05:31, 23.67s/it] {'loss': '0.5692', 'grad_norm': '0.04432', 'learning_rate': '0', 'ppl': '1.767', 'memory/max_active (GiB)': '45.83', 'memory/max_allocated (GiB)': '45.83', 'memory/device_reserved (GiB)': '51.3', 'tokens/train_per_sec_per_gpu': '200.9', 'tokens/total': 262144, 'tokens/trainable': 42059, 'epoch': '0.0708'}
	7%\|██████▊ \| 1/15 [00:23<05:31, 23.67s/it] 13%\|█████████████▋ \| 2/15 [00:38<03:58, 18.36s/it] {'loss': '0.6769', 'grad_norm': '0.04985', 'learning_rate': '2e-05', 'ppl': '1.968', 'memory/max_active (GiB)': '46.14', 'memory/max_allocated (GiB)': '46.14', 'memory/device_reserved (GiB)': '51.31', 'tokens/train_per_sec_per_gpu': '146.2', 'tokens/total': 524288, 'tokens/trainable': 81290, 'epoch': '0.1416'}
	13%\|█████████████▋ \| 2/15 [00:38<03:58, 18.36s/it] 20%\|████████████████████▌ \| 3/15 [00:53<03:21, 16.80s/it] {'loss': '0.6294', 'grad_norm': '0.04196', 'learning_rate': '4e-05', 'ppl': '1.876', 'memory/max_active (GiB)': '46.14', 'memory/max_allocated (GiB)': '46.14', 'memory/device_reserved (GiB)': '51.31', 'tokens/train_per_sec_per_gpu': '326.4', 'tokens/total': 786432, 'tokens/trainable': 127458, 'epoch': '0.2124'}
	20%\|████████████████████▌ \| 3/15 [00:53<03:21, 16.80s/it] 27%\|███████████████████████████▍ \| 4/15 [01:08<02:56, 16.04s/it] {'loss': '0.6184', 'grad_norm': '0.04567', 'learning_rate': '3.942e-05', 'ppl': '1.856', 'memory/max_active (GiB)': '46.14', 'memory/max_allocated (GiB)': '46.14', 'memory/device_reserved (GiB)': '51.31', 'tokens/train_per_sec_per_gpu': '159', 'tokens/total': 1048576, 'tokens/trainable': 169289, 'epoch': '0.2832'}
	27%\|███████████████████████████▍ \| 4/15 [01:08<02:56, 16.04s/it] 33%\|██████████████████████████████████▎ \| 5/15 [01:22<02:35, 15.55s/it] {'loss': '0.585', 'grad_norm': '0.05341', 'learning_rate': '3.771e-05', 'ppl': '1.795', 'memory/max_active (GiB)': '46.14', 'memory/max_allocated (GiB)': '46.14', 'memory/device_reserved (GiB)': '51.31', 'tokens/train_per_sec_per_gpu': '228.4', 'tokens/total': 1310720, 'tokens/trainable': 208301, 'epoch': '0.354'}
	33%\|██████████████████████████████████▎ \| 5/15 [01:22<02:35, 15.55s/it][2026-04-21 00:46:26,235] [INFO] [axolotl.core.trainers.base.evaluate:410] [PID:4071480] Running evaluation step...

	0%\| \| 0/13 [00:00<?, ?it/s][A
	15%\|███████████████▊ \| 2/13 [00:00<00:03, 3.58it/s][A
	23%\|███████████████████████▊ \| 3/13 [00:01<00:03, 2.52it/s][A
	31%\|███████████████████████████████▋ \| 4/13 [00:01<00:04, 2.15it/s][A
	38%\|███████████████████████████████████████▌ \| 5/13 [00:02<00:03, 2.01it/s][A
	46%\|███████████████████████████████████████████████▌ \| 6/13 [00:02<00:03, 1.93it/s][A
	54%\|███████████████████████████████████████████████████████▍ \| 7/13 [00:03<00:03, 1.88it/s][A
	62%\|███████████████████████████████████████████████████████████████▍ \| 8/13 [00:03<00:02, 1.85it/s][A
	69%\|███████████████████████████████████████████████████████████████████████▎ \| 9/13 [00:04<00:02, 1.76it/s][A
	77%\|██████████████████████████████████████████████████████████████████████████████▍ \| 10/13 [00:05<00:01, 1.78it/s][A
	85%\|██████████████████████████████████████████████████████████████████████████████████████▎ \| 11/13 [00:05<00:01, 1.77it/s][A
	92%\|██████████████████████████████████████████████████████████████████████████████████████████████▏ \| 12/13 [00:06<00:00, 1.76it/s][ATraceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-lcqdh4im'
	Traceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-rj17bopi'
	Traceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-nohm303u'
	Traceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-oqox9irv'

	100%\|██████████████████████████████████████████████████████████████████████████████████████████████████████\| 13/13 [00:07<00:00, 1.61it/s][A
	[A{'eval_loss': '0.6517', 'eval_runtime': '7.71', 'eval_samples_per_second': '12.97', 'eval_steps_per_second': '1.686', 'eval_ppl': '1.919', 'memory/max_active (GiB)': '38.19', 'memory/max_allocated (GiB)': '38.19', 'memory/device_reserved (GiB)': '51.31', 'epoch': '0.354', 'tokens/train_per_sec_per_gpu': '0'}
	33%\|██████████████████████████████████▎ \| 5/15 [01:30<02:35, 15.55s/it]
	100%\|██████████████████████████████████████████████████████████████████████████████████████████████████████\| 13/13 [00:07<00:00, 1.61it/s][A
	[A[2026-04-21 00:46:34,075] [INFO] [axolotl.core.trainers.base._save:741] [PID:4071480] Saving model checkpoint to /home/tkwang/scratch/SecSteer-v2/axolotl-outputs/lora/Qwen2.5-Coder-7B-func-stage1/checkpoint-5
	[2026-04-21 00:46:35,144] [WARNING] [py.warnings._showwarnmsg:112] [PID:4071480] /scratch/tkwang/SecSteer-v2/.venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
	return func(args, *kwargs)

	40%\|█████████████████████████████████████████▏ \| 6/15 [01:46<02:46, 18.48s/it] {'loss': '0.5953', 'grad_norm': '0.06175', 'learning_rate': '3.497e-05', 'ppl': '1.814', 'memory/max_active (GiB)': '46.14', 'memory/max_allocated (GiB)': '46.14', 'memory/device_reserved (GiB)': '52.46', 'tokens/train_per_sec_per_gpu': '146.2', 'tokens/total': 1572864, 'tokens/trainable': 241818, 'epoch': '0.4248'}
	40%\|█████████████████████████████████████████▏ \| 6/15 [01:46<02:46, 18.48s/it] 47%\|████████████████████████████████████████████████ \| 7/15 [02:01<02:17, 17.23s/it] {'loss': '0.6125', 'grad_norm': '0.06692', 'learning_rate': '3.136e-05', 'ppl': '1.845', 'memory/max_active (GiB)': '46.14', 'memory/max_allocated (GiB)': '46.14', 'memory/device_reserved (GiB)': '52.46', 'tokens/train_per_sec_per_gpu': '114', 'tokens/total': 1835008, 'tokens/trainable': 276845, 'epoch': '0.4956'}
	47%\|████████████████████████████████████████████████ \| 7/15 [02:01<02:17, 17.23s/it] 53%\|██████████████████████████████████████████████████████▉ \| 8/15 [02:16<01:55, 16.46s/it] {'loss': '0.6185', 'grad_norm': '0.06703', 'learning_rate': '2.709e-05', 'ppl': '1.856', 'memory/max_active (GiB)': '46.14', 'memory/max_allocated (GiB)': '46.14', 'memory/device_reserved (GiB)': '52.47', 'tokens/train_per_sec_per_gpu': '119.3', 'tokens/total': 2097152, 'tokens/trainable': 313509, 'epoch': '0.5664'}
	53%\|██████████████████████████████████████████████████████▉ \| 8/15 [02:16<01:55, 16.46s/it] 60%\|█████████████████████████████████████████████████████████████▊ \| 9/15 [02:31<01:35, 15.93s/it] {'loss': '0.5976', 'grad_norm': '0.06155', 'learning_rate': '2.241e-05', 'ppl': '1.818', 'memory/max_active (GiB)': '46.14', 'memory/max_allocated (GiB)': '46.14', 'memory/device_reserved (GiB)': '52.47', 'tokens/train_per_sec_per_gpu': '112.7', 'tokens/total': 2359296, 'tokens/trainable': 355063, 'epoch': '0.6372'}
	60%\|█████████████████████████████████████████████████████████████▊ \| 9/15 [02:31<01:35, 15.93s/it] 67%\|████████████████████████████████████████████████████████████████████ \| 10/15 [02:45<01:17, 15.54s/it] {'loss': '0.6071', 'grad_norm': '0.07197', 'learning_rate': '1.759e-05', 'ppl': '1.835', 'memory/max_active (GiB)': '46.14', 'memory/max_allocated (GiB)': '46.14', 'memory/device_reserved (GiB)': '52.47', 'tokens/train_per_sec_per_gpu': '163.8', 'tokens/total': 2621440, 'tokens/trainable': 389457, 'epoch': '0.708'}
	67%\|████████████████████████████████████████████████████████████████████ \| 10/15 [02:45<01:17, 15.54s/it][2026-04-21 00:47:49,311] [INFO] [axolotl.core.trainers.base.evaluate:410] [PID:4071480] Running evaluation step...

	0%\| \| 0/13 [00:00<?, ?it/s][A
	15%\|███████████████▊ \| 2/13 [00:00<00:03, 3.60it/s][A
	23%\|███████████████████████▊ \| 3/13 [00:01<00:04, 2.50it/s][A
	31%\|███████████████████████████████▋ \| 4/13 [00:01<00:04, 2.13it/s][A
	38%\|███████████████████████████████████████▌ \| 5/13 [00:02<00:04, 2.00it/s][A
	46%\|███████████████████████████████████████████████▌ \| 6/13 [00:02<00:03, 1.92it/s][A
	54%\|███████████████████████████████████████████████████████▍ \| 7/13 [00:03<00:03, 1.87it/s][A
	62%\|███████████████████████████████████████████████████████████████▍ \| 8/13 [00:03<00:02, 1.84it/s][A
	69%\|███████████████████████████████████████████████████████████████████████▎ \| 9/13 [00:04<00:02, 1.75it/s][A
	77%\|██████████████████████████████████████████████████████████████████████████████▍ \| 10/13 [00:05<00:01, 1.78it/s][A
	85%\|██████████████████████████████████████████████████████████████████████████████████████▎ \| 11/13 [00:05<00:01, 1.76it/s][A
	92%\|██████████████████████████████████████████████████████████████████████████████████████████████▏ \| 12/13 [00:06<00:00, 1.76it/s][ATraceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-gki_d3e8'
	Traceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-g5405drs'
	Traceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-1_u4h44m'
	Traceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-z_8a8qw2'

	100%\|██████████████████████████████████████████████████████████████████████████████████████████████████████\| 13/13 [00:06<00:00, 1.70it/s][A
	[A{'eval_loss': '0.6357', 'eval_runtime': '7.615', 'eval_samples_per_second': '13.13', 'eval_steps_per_second': '1.707', 'eval_ppl': '1.888', 'memory/max_active (GiB)': '38.19', 'memory/max_allocated (GiB)': '38.19', 'memory/device_reserved (GiB)': '52.47', 'epoch': '0.708', 'tokens/train_per_sec_per_gpu': '0'}
	67%\|████████████████████████████████████████████████████████████████████ \| 10/15 [02:53<01:17, 15.54s/it]
	100%\|██████████████████████████████████████████████████████████████████████████████████████████████████████\| 13/13 [00:06<00:00, 1.70it/s][A
	[A[2026-04-21 00:47:56,986] [INFO] [axolotl.core.trainers.base._save:741] [PID:4071480] Saving model checkpoint to /home/tkwang/scratch/SecSteer-v2/axolotl-outputs/lora/Qwen2.5-Coder-7B-func-stage1/checkpoint-10
	[2026-04-21 00:47:57,836] [WARNING] [py.warnings._showwarnmsg:112] [PID:4071480] /scratch/tkwang/SecSteer-v2/.venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
	return func(args, *kwargs)

	73%\|██████████████████████████████████████████████████████████████████████████▊ \| 11/15 [03:10<01:12, 18.17s/it] {'loss': '0.6194', 'grad_norm': '0.06332', 'learning_rate': '1.291e-05', 'ppl': '1.858', 'memory/max_active (GiB)': '46.14', 'memory/max_allocated (GiB)': '46.14', 'memory/device_reserved (GiB)': '52.46', 'tokens/train_per_sec_per_gpu': '133.4', 'tokens/total': 2883584, 'tokens/trainable': 430903, 'epoch': '0.7788'}
	73%\|██████████████████████████████████████████████████████████████████████████▊ \| 11/15 [03:10<01:12, 18.17s/it] 80%\|█████████████████████████████████████████████████████████████████████████████████▌ \| 12/15 [03:24<00:51, 17.14s/it] {'loss': '0.5998', 'grad_norm': '0.06604', 'learning_rate': '8.639e-06', 'ppl': '1.822', 'memory/max_active (GiB)': '46.14', 'memory/max_allocated (GiB)': '46.14', 'memory/device_reserved (GiB)': '52.47', 'tokens/train_per_sec_per_gpu': '179.2', 'tokens/total': 3145728, 'tokens/trainable': 464302, 'epoch': '0.8496'}
	80%\|█████████████████████████████████████████████████████████████████████████████████▌ \| 12/15 [03:24<00:51, 17.14s/it] 87%\|████████████████████████████████████████████████████████████████████████████████████████▍ \| 13/15 [03:39<00:32, 16.41s/it] {'loss': '0.6488', 'grad_norm': '0.05788', 'learning_rate': '5.03e-06', 'ppl': '1.913', 'memory/max_active (GiB)': '46.14', 'memory/max_allocated (GiB)': '46.14', 'memory/device_reserved (GiB)': '52.47', 'tokens/train_per_sec_per_gpu': '186', 'tokens/total': 3407872, 'tokens/trainable': 500768, 'epoch': '0.9204'}
	87%\|████████████████████████████████████████████████████████████████████████████████████████▍ \| 13/15 [03:39<00:32, 16.41s/it] 93%\|███████████████████████████████████████████████████████████████████████████████████████████████▏ \| 14/15 [03:54<00:15, 15.89s/it] {'loss': '0.5921', 'grad_norm': '0.05555', 'learning_rate': '2.291e-06', 'ppl': '1.808', 'memory/max_active (GiB)': '46.14', 'memory/max_allocated (GiB)': '46.14', 'memory/device_reserved (GiB)': '52.47', 'tokens/train_per_sec_per_gpu': '199.4', 'tokens/total': 3670016, 'tokens/trainable': 535604, 'epoch': '0.9912'}
	93%\|███████████████████████████████████████████████████████████████████████████████████████████████▏ \| 14/15 [03:54<00:15, 15.89s/it]Traceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-x7zy31s2'
	Traceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-jya944mu'
	Traceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-_ectlwho'
	Traceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-ev6gxov2'
	100%\|██████████████████████████████████████████████████████████████████████████████████████████████████████\| 15/15 [03:56<00:00, 11.71s/it] {'loss': '0.6697', 'grad_norm': '0.08438', 'learning_rate': '5.812e-07', 'ppl': '1.954', 'memory/max_active (GiB)': '45.98', 'memory/max_allocated (GiB)': '45.98', 'memory/device_reserved (GiB)': '52.47', 'tokens/train_per_sec_per_gpu': '1772', 'tokens/total': 3702784, 'tokens/trainable': 542440, 'epoch': '1'}
	100%\|██████████████████████████████████████████████████████████████████████████████████████████████████████\| 15/15 [03:56<00:00, 11.71s/it][2026-04-21 00:48:59,675] [INFO] [axolotl.core.trainers.base.evaluate:410] [PID:4071480] Running evaluation step...

	0%\| \| 0/13 [00:00<?, ?it/s][A
	15%\|███████████████▊ \| 2/13 [00:00<00:03, 3.59it/s][A
	23%\|███████████████████████▊ \| 3/13 [00:01<00:03, 2.51it/s][A
	31%\|███████████████████████████████▋ \| 4/13 [00:01<00:04, 2.14it/s][A
	38%\|███████████████████████████████████████▌ \| 5/13 [00:02<00:03, 2.01it/s][A
	46%\|███████████████████████████████████████████████▌ \| 6/13 [00:02<00:03, 1.93it/s][A
	54%\|███████████████████████████████████████████████████████▍ \| 7/13 [00:03<00:03, 1.88it/s][A
	62%\|███████████████████████████████████████████████████████████████▍ \| 8/13 [00:03<00:02, 1.84it/s][A
	69%\|███████████████████████████████████████████████████████████████████████▎ \| 9/13 [00:04<00:02, 1.76it/s][A
	77%\|██████████████████████████████████████████████████████████████████████████████▍ \| 10/13 [00:05<00:01, 1.77it/s][A
	85%\|██████████████████████████████████████████████████████████████████████████████████████▎ \| 11/13 [00:05<00:01, 1.76it/s][A
	92%\|██████████████████████████████████████████████████████████████████████████████████████████████▏ \| 12/13 [00:06<00:00, 1.76it/s][ATraceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-8yrj3uhp'
	Traceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-cs3lmw11'
	Traceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-lm37mbe5'
	Traceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-1s3lsuyi'

	100%\|██████████████████████████████████████████████████████████████████████████████████████████████████████\| 13/13 [00:06<00:00, 1.71it/s][A
	[A{'eval_loss': '0.6311', 'eval_runtime': '7.613', 'eval_samples_per_second': '13.13', 'eval_steps_per_second': '1.708', 'eval_ppl': '1.88', 'memory/max_active (GiB)': '38.19', 'memory/max_allocated (GiB)': '38.19', 'memory/device_reserved (GiB)': '52.47', 'epoch': '1', 'tokens/train_per_sec_per_gpu': '0'}
	100%\|██████████████████████████████████████████████████████████████████████████████████████████████████████\| 15/15 [04:03<00:00, 11.71s/it]
	100%\|██████████████████████████████████████████████████████████████████████████████████████████████████████\| 13/13 [00:06<00:00, 1.71it/s][A
	[A[2026-04-21 00:49:07,404] [INFO] [axolotl.core.trainers.base._save:741] [PID:4071480] Saving model checkpoint to /home/tkwang/scratch/SecSteer-v2/axolotl-outputs/lora/Qwen2.5-Coder-7B-func-stage1/checkpoint-15
	[2026-04-21 00:49:08,489] [WARNING] [py.warnings._showwarnmsg:112] [PID:4071480] /scratch/tkwang/SecSteer-v2/.venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
	return func(args, *kwargs)

	{'train_runtime': '249.3', 'train_samples_per_second': '3.85', 'train_steps_per_second': '0.06', 'train_loss': '0.616', 'memory/max_active (GiB)': '14.86', 'memory/max_allocated (GiB)': '14.86', 'memory/device_reserved (GiB)': '43.12', 'epoch': '1', 'tokens/train_per_sec_per_gpu': '0'}
	100%\|██████████████████████████████████████████████████████████████████████████████████████████████████████\| 15/15 [04:05<00:00, 11.71s/it] 100%\|██████████████████████████████████████████████████████████████████████████████████████████████████████\| 15/15 [04:06<00:00, 16.45s/it]
	[2026-04-21 00:49:10,233] [INFO] [axolotl.train.save_trained_model:241] [PID:4071480] Training completed! Saving trained model to /home/tkwang/scratch/SecSteer-v2/axolotl-outputs/lora/Qwen2.5-Coder-7B-func-stage1.
	[2026-04-21 00:49:11,013] [INFO] [axolotl.train.save_trained_model:355] [PID:4071480] Model successfully saved to /home/tkwang/scratch/SecSteer-v2/axolotl-outputs/lora/Qwen2.5-Coder-7B-func-stage1