| 2026-01-14 00:36:01 | INFO | espnet3 | === ESPnet3 run started: 2026-01-14T00:36:01.044208 === | |
| 2026-01-14 00:36:01 | INFO | espnet3 | Command: /data/user_data/msomeki/espnet3/.venv/bin/python run.py --stage pack_model --publish_config conf/publish.yaml --train_config conf/train.yaml | |
| 2026-01-14 00:36:01 | INFO | espnet3 | Python: 3.11.13 (main, Aug 18 2025, 19:19:13) [Clang 20.1.4 ] | |
| 2026-01-14 00:36:01 | INFO | espnet3 | Working directory: /home/msomeki/00_systems/espnet3/egs3/mini_an4/asr | |
| 2026-01-14 00:36:01 | INFO | espnet3 | train config: /home/msomeki/00_systems/espnet3/egs3/mini_an4/asr/conf/train_asr_rnn_data_aug_debug.yaml | |
| 2026-01-14 00:36:01 | INFO | espnet3 | publish config: /home/msomeki/00_systems/espnet3/egs3/mini_an4/asr/conf/publish.yaml | |
| 2026-01-14 00:36:01 | INFO | espnet3 | Git: commit=8509faad9811b58d5024f29fb9d68ffb026b5e73, short_commit=8509faad9, branch=espnet3/recipe/asr_ls100, worktree=dirty | |
| 2026-01-14 00:36:01 | INFO | espnet3 | Cluster env: | |
| OMPI_MCA_plm_slurm_args=--external-launcher | |
| SLURM_CLUSTER_NAME=babel | |
| SLURM_CONF=/var/spool/slurmd/conf-cache/slurm.conf | |
| SLURM_CPUS_ON_NODE=1 | |
| SLURM_CPUS_PER_TASK=1 | |
| SLURM_CPU_BIND=quiet,mask_cpu:0x0000000000010000 | |
| SLURM_CPU_BIND_LIST=0x0000000000010000 | |
| SLURM_CPU_BIND_TYPE=mask_cpu: | |
| SLURM_CPU_BIND_VERBOSE=quiet | |
| SLURM_DISTRIBUTION=cyclic,pack | |
| SLURM_GTIDS=0 | |
| SLURM_JOBID=6122041 | |
| SLURM_JOB_ACCOUNT=swatanab | |
| SLURM_JOB_CPUS_PER_NODE=1 | |
| SLURM_JOB_END_TIME=1768401875 | |
| SLURM_JOB_GID=2709140 | |
| SLURM_JOB_GROUP=msomeki | |
| SLURM_JOB_ID=6122041 | |
| SLURM_JOB_NAME=bash | |
| SLURM_JOB_NODELIST=babel-o9-16 | |
| SLURM_JOB_NUM_NODES=1 | |
| SLURM_JOB_PARTITION=debug | |
| SLURM_JOB_QOS=debug_qos | |
| SLURM_JOB_START_TIME=1768358675 | |
| SLURM_JOB_UID=2709140 | |
| SLURM_JOB_USER=msomeki | |
| SLURM_LAUNCH_NODE_IPADDR=172.16.1.2 | |
| SLURM_LOCALID=0 | |
| SLURM_MEM_PER_NODE=4096 | |
| SLURM_NNODES=1 | |
| SLURM_NODEID=0 | |
| SLURM_NODELIST=babel-o9-16 | |
| SLURM_NPROCS=1 | |
| SLURM_NTASKS=1 | |
| SLURM_NTASKS_PER_NODE=1 | |
| SLURM_PRIO_PROCESS=0 | |
| SLURM_PROCID=0 | |
| SLURM_PTY_PORT=40465 | |
| SLURM_PTY_WIN_COL=112 | |
| SLURM_PTY_WIN_ROW=61 | |
| SLURM_SCRIPT_CONTEXT=prolog_task | |
| SLURM_SRUN_COMM_HOST=172.16.1.2 | |
| SLURM_SRUN_COMM_PORT=33789 | |
| SLURM_STEPID=0 | |
| SLURM_STEP_ID=0 | |
| SLURM_STEP_LAUNCHER_PORT=33789 | |
| SLURM_STEP_NODELIST=babel-o9-16 | |
| SLURM_STEP_NUM_NODES=1 | |
| SLURM_STEP_NUM_TASKS=1 | |
| SLURM_STEP_TASKS_PER_NODE=1 | |
| SLURM_SUBMIT_DIR=/home/msomeki/00_systems/espnet3 | |
| SLURM_SUBMIT_HOST=login1 | |
| SLURM_TASKS_PER_NODE=1 | |
| SLURM_TASK_PID=3334910 | |
| SLURM_TOPOLOGY_ADDR=babel-o9-16 | |
| SLURM_TOPOLOGY_ADDR_PATTERN=node | |
| SLURM_TRES_PER_TASK=cpu=1 | |
| SLURM_UMASK=0027 | |
| 2026-01-14 00:36:01 | INFO | espnet3 | Runtime env: | |
| LD_LIBRARY_PATH=/home/msomeki/00_systems/espnet3/tools/espeak-ng/lib:/home/msomeki/00_systems/espnet3/tools/lib:/home/msomeki/00_systems/espnet3/tools/lib64:/home/msomeki/00_systems/espnet3/tools/espeak-ng/lib:/home/msomeki/00_systems/espnet3/tools/lib:/home/msomeki/00_systems/espnet3/tools/lib64:/home/msomeki/00_systems/espnet3/tools/espeak-ng/lib:/home/msomeki/00_systems/espnet3/tools/lib:/home/msomeki/00_systems/espnet3/tools/lib64:/home/msomeki/00_systems/espnet3/tools/espeak-ng/lib:/home/msomeki/00_systems/espnet3/tools/lib:/home/msomeki/00_systems/espnet3/tools/lib64:/home/msomeki/00_systems/espnet3/tools/espeak-ng/lib:/home/msomeki/00_systems/espnet3/tools/lib:/home/msomeki/00_systems/espnet3/tools/lib64:/home/msomeki/00_systems/espnet3/tools/espeak-ng/lib:/home/msomeki/00_systems/espnet3/tools/lib:/home/msomeki/00_systems/espnet3/tools/lib64:/home/msomeki/00_systems/espnet3/tools/espeak-ng/lib:/home/msomeki/00_systems/espnet3/tools/lib:/home/msomeki/00_systems/espnet3/tools/lib64:/home/msomeki/00_systems/espnet3/tools/espeak-ng/lib:/home/msomeki/00_systems/espnet3/tools/lib:/home/msomeki/00_systems/espnet3/tools/lib64:/home/msomeki/00_systems/espnet3/tools/espeak-ng/lib:/home/msomeki/00_systems/espnet3/tools/lib:/home/msomeki/00_systems/espnet3/tools/lib64:/home/msomeki/00_systems/espnet3/tools/espeak-ng/lib:/home/msomeki/00_systems/espnet3/tools/lib:/home/msomeki/00_systems/espnet3/tools/lib64:/home/msomeki/00_systems/espnet3/tools/espeak-ng/lib:/home/msomeki/00_systems/espnet3/tools/lib:/home/msomeki/00_systems/espnet3/tools/lib64:/home/msomeki/00_systems/espnet3/tools/espeak-ng/lib:/home/msomeki/00_systems/espnet3/tools/lib:/home/msomeki/00_systems/espnet3/tools/lib64:/home/msomeki/00_systems/espnet3/tools/espeak-ng/lib:/home/msomeki/00_systems/espnet3/tools/lib:/home/msomeki/00_systems/espnet3/tools/lib64:/home/msomeki/00_systems/espnet3/tools/espeak-ng/lib:/home/msomeki/00_systems/espnet3/tools/lib:/home/msomeki/00_systems/espnet3/tools/lib64:/home/msomeki/00_systems/espnet3/tools/espeak-ng/lib:/home/msomeki/00_systems/espnet3/tools/lib:/home/msomeki/00_systems/espnet3/tools/lib64: | |
| PATH=/home/msomeki/00_systems/espnet3/tools/ffmpeg-release:/home/msomeki/00_systems/espnet3/tools/festival/bin:/home/msomeki/00_systems/espnet3/tools/MBROLA/Bin:/home/msomeki/00_systems/espnet3/tools/espeak-ng/bin:/home/msomeki/00_systems/espnet3/tools/BeamformIt:/home/msomeki/00_systems/espnet3/tools/kenlm/build/bin:/home/msomeki/00_systems/espnet3/tools/PESQ/P862_annex_A_2005_CD/source:/home/msomeki/00_systems/espnet3/tools/nkf/nkf-2.1.4:/home/msomeki/00_systems/espnet3/tools/moses/scripts/tokenizer:/home/msomeki/00_systems/espnet3/tools/moses/scripts/generic:/home/msomeki/00_systems/espnet3/tools/tools/moses/scripts/recaser:/home/msomeki/00_systems/espnet3/tools/moses/scripts/training:/home/msomeki/00_systems/espnet3/tools/mwerSegmenter:/home/msomeki/00_systems/espnet3/tools/sctk/bin:/home/msomeki/00_systems/espnet3/tools/sph2pipe:/home/msomeki/00_systems/espnet3/tools/sentencepiece_commands:/data/user_data/msomeki/espnet3/.venv/bin:/home/msomeki/.pixi/bin:/home/msomeki/local/bin:/home/msomeki/utils:/usr/share/Modules/bin:/home/msomeki/00_systems/espnet3/tools/ffmpeg-release:/home/msomeki/00_systems/espnet3/tools/festival/bin:/home/msomeki/00_systems/espnet3/tools/MBROLA/Bin:/home/msomeki/00_systems/espnet3/tools/espeak-ng/bin:/home/msomeki/00_systems/espnet3/tools/BeamformIt:/home/msomeki/00_systems/espnet3/tools/kenlm/build/bin:/home/msomeki/00_systems/espnet3/tools/PESQ/P862_annex_A_2005_CD/source:/home/msomeki/00_systems/espnet3/tools/nkf/nkf-2.1.4:/home/msomeki/00_systems/espnet3/tools/moses/scripts/tokenizer:/home/msomeki/00_systems/espnet3/tools/moses/scripts/generic:/home/msomeki/00_systems/espnet3/tools/tools/moses/scripts/recaser:/home/msomeki/00_systems/espnet3/tools/moses/scripts/training:/home/msomeki/00_systems/espnet3/tools/mwerSegmenter:/home/msomeki/00_systems/espnet3/tools/sctk/bin:/home/msomeki/00_systems/espnet3/tools/sph2pipe:/home/msomeki/00_systems/espnet3/tools/sentencepiece_commands:/home/msomeki/00_systems/espnet3/tools/ffmpeg-release:/home/msomeki/00_systems/espnet3/tools/festival/bin:/home/msomeki/00_systems/espnet3/tools/MBROLA/Bin:/home/msomeki/00_systems/espnet3/tools/espeak-ng/bin:/home/msomeki/00_systems/espnet3/tools/BeamformIt:/home/msomeki/00_systems/espnet3/tools/kenlm/build/bin:/home/msomeki/00_systems/espnet3/tools/PESQ/P862_annex_A_2005_CD/source:/home/msomeki/00_systems/espnet3/tools/nkf/nkf-2.1.4:/home/msomeki/00_systems/espnet3/tools/moses/scripts/tokenizer:/home/msomeki/00_systems/espnet3/tools/moses/scripts/generic:/home/msomeki/00_systems/espnet3/tools/tools/moses/scripts/recaser:/home/msomeki/00_systems/espnet3/tools/moses/scripts/training:/home/msomeki/00_systems/espnet3/tools/mwerSegmenter:/home/msomeki/00_systems/espnet3/tools/sctk/bin:/home/msomeki/00_systems/espnet3/tools/sph2pipe:/home/msomeki/00_systems/espnet3/tools/sentencepiece_commands:/home/msomeki/.pixi/bin:/home/msomeki/local/bin:/home/msomeki/utils:/home/msomeki/.local/bin:/home/msomeki/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin | |
| PYTHONPATH=/home/msomeki/00_systems/espnet3/tools/RawNet/python/RawNet3:/home/msomeki/00_systems/espnet3/tools/RawNet/python/RawNet3/models:../../../:../../TEMPLATE/asr:/home/msomeki/00_systems/espnet3/egs3/mini_an4/asr:/home/msomeki/00_systems/espnet3/tools/RawNet/python/RawNet3:/home/msomeki/00_systems/espnet3/tools/RawNet/python/RawNet3/models:/home/msomeki/00_systems/espnet3/tools/RawNet/python/RawNet3:/home/msomeki/00_systems/espnet3/tools/RawNet/python/RawNet3/models:../../../:../../TEMPLATE/asr:/home/msomeki/00_systems/espnet3/egs3/mini_an4/asr:/home/msomeki/00_systems/espnet3/tools/RawNet/python/RawNet3:/home/msomeki/00_systems/espnet3/tools/RawNet/python/RawNet3/models:/home/msomeki/00_systems/espnet3/tools/RawNet/python/RawNet3:/home/msomeki/00_systems/espnet3/tools/RawNet/python/RawNet3/models:../../../:../../TEMPLATE/asr:/home/msomeki/00_systems/espnet3/egs3/mini_an4/asr:/home/msomeki/00_systems/espnet3/tools/RawNet/python/RawNet3:/home/msomeki/00_systems/espnet3/tools/RawNet/python/RawNet3/models:/home/msomeki/00_systems/espnet3/tools/RawNet/python/RawNet3:/home/msomeki/00_systems/espnet3/tools/RawNet/python/RawNet3/models:../../../:../../TEMPLATE/asr:/home/msomeki/00_systems/espnet3/egs3/mini_an4/asr:/home/msomeki/00_systems/espnet3/tools/RawNet/python/RawNet3:/home/msomeki/00_systems/espnet3/tools/RawNet/python/RawNet3/models:/home/msomeki/00_systems/espnet3/tools/RawNet/python/RawNet3:/home/msomeki/00_systems/espnet3/tools/RawNet/python/RawNet3/models:../../../:../../TEMPLATE/asr:/home/msomeki/00_systems/espnet3/egs3/mini_an4/asr:/home/msomeki/00_systems/espnet3/tools/RawNet/python/RawNet3:/home/msomeki/00_systems/espnet3/tools/RawNet/python/RawNet3/models:/home/msomeki/00_systems/espnet3/tools/RawNet/python/RawNet3:/home/msomeki/00_systems/espnet3/tools/RawNet/python/RawNet3/models:/home/msomeki/00_systems/espnet3/tools/RawNet/python/RawNet3:/home/msomeki/00_systems/espnet3/tools/RawNet/python/RawNet3/models:../../../:../../TEMPLATE/asr:/home/msomeki/00_systems/espnet3/egs3/mini_an4/asr:/home/msomeki/00_systems/espnet3/tools/RawNet/python/RawNet3:/home/msomeki/00_systems/espnet3/tools/RawNet/python/RawNet3/models:/home/msomeki/00_systems/espnet3/tools/RawNet/python/RawNet3:/home/msomeki/00_systems/espnet3/tools/RawNet/python/RawNet3/models:../../../:../../TEMPLATE/asr:/home/msomeki/00_systems/espnet3/egs3/mini_an4/asr:/home/msomeki/00_systems/espnet3/tools/RawNet/python/RawNet3:/home/msomeki/00_systems/espnet3/tools/RawNet/python/RawNet3/models: | |
| 2026-01-14 00:36:01 | INFO | espnet3 | Train config content: | |
| num_device: 1 | |
| num_nodes: 1 | |
| task: espnet3.systems.asr.task.ASRTask | |
| recipe_dir: . | |
| data_dir: ./data | |
| exp_tag: train_asr_rnn_data_aug_debug | |
| exp_dir: ./exp/train_asr_rnn_data_aug_debug | |
| stats_dir: ./exp/stats | |
| decode_dir: ./exp/train_asr_rnn_data_aug_debug/decode | |
| dataset_dir: ./data/mini_an4 | |
| create_dataset: | |
| func: src.create_dataset.create_dataset | |
| dataset_dir: ./data/mini_an4 | |
| archive_path: ./../../egs2/mini_an4/asr1/downloads.tar.gz | |
| dataset: | |
| _target_: espnet3.components.data.data_organizer.DataOrganizer | |
| train: | |
| - name: train_nodev | |
| dataset: | |
| _target_: src.dataset.MiniAN4Dataset | |
| manifest_path: ./data/mini_an4/manifest/train_nodev.tsv | |
| valid: | |
| - name: train_dev | |
| dataset: | |
| _target_: src.dataset.MiniAN4Dataset | |
| manifest_path: ./data/mini_an4/manifest/train_dev.tsv | |
| preprocessor: | |
| _target_: espnet2.train.preprocessor.CommonPreprocessor | |
| _convert_: all | |
| fs: 16000 | |
| train: true | |
| data_aug_effects: | |
| - - 0.1 | |
| - contrast | |
| - enhancement_amount: 75.0 | |
| - - 0.1 | |
| - highpass | |
| - cutoff_freq: 5000 | |
| Q: 0.707 | |
| - - 0.1 | |
| - equalization | |
| - center_freq: 1000 | |
| gain: 0 | |
| Q: 0.707 | |
| - - 0.1 | |
| - - - 0.3 | |
| - speed_perturb | |
| - factor: 0.9 | |
| - - 0.3 | |
| - speed_perturb | |
| - factor: 1.1 | |
| - - 0.3 | |
| - speed_perturb | |
| - factor: 1.3 | |
| data_aug_num: | |
| - 1 | |
| - 4 | |
| data_aug_prob: 1.0 | |
| token_type: bpe | |
| token_list: ./data/bpe_30/tokens.txt | |
| bpemodel: ./data/bpe_30/bpe.model | |
| parallel: | |
| env: local | |
| n_workers: 1 | |
| dataloader: | |
| collate_fn: | |
| _target_: espnet2.train.collate_fn.CommonCollateFn | |
| int_pad_value: -1 | |
| train: | |
| multiple_iterator: false | |
| num_shards: 1 | |
| iter_factory: | |
| _target_: espnet2.iterators.sequence_iter_factory.SequenceIterFactory | |
| shuffle: true | |
| collate_fn: | |
| _target_: espnet2.train.collate_fn.CommonCollateFn | |
| int_pad_value: -1 | |
| num_workers: 0 | |
| batches: | |
| type: sorted | |
| shape_files: | |
| - ./exp/stats/train/feats_shape | |
| batch_size: 2 | |
| batch_bins: 200000 | |
| valid: | |
| multiple_iterator: false | |
| num_shards: 1 | |
| iter_factory: | |
| _target_: espnet2.iterators.sequence_iter_factory.SequenceIterFactory | |
| shuffle: false | |
| collate_fn: | |
| _target_: espnet2.train.collate_fn.CommonCollateFn | |
| int_pad_value: -1 | |
| batches: | |
| type: sorted | |
| shape_files: | |
| - ./exp/stats/valid/feats_shape | |
| batch_size: 2 | |
| batch_bins: 200000 | |
| optim: | |
| _target_: torch.optim.Adam | |
| lr: 0.001 | |
| weight_decay: 0.0 | |
| scheduler: | |
| _target_: torch.optim.lr_scheduler.ReduceLROnPlateau | |
| mode: min | |
| factor: 0.5 | |
| patience: 1 | |
| val_scheduler_criterion: valid/loss | |
| best_model_criterion: | |
| - - valid/acc | |
| - 1 | |
| - max | |
| trainer: | |
| accelerator: auto | |
| devices: 1 | |
| num_nodes: 1 | |
| accumulate_grad_batches: 1 | |
| check_val_every_n_epoch: 1 | |
| gradient_clip_val: 1.0 | |
| log_every_n_steps: 1 | |
| max_epochs: 1 | |
| limit_train_batches: 1 | |
| limit_val_batches: 1 | |
| precision: 32 | |
| logger: | |
| - _target_: lightning.pytorch.loggers.TensorBoardLogger | |
| save_dir: ./exp/train_asr_rnn_data_aug_debug/tensorboard | |
| name: tb_logger | |
| strategy: auto | |
| tokenizer: | |
| vocab_size: 30 | |
| character_coverage: 1.0 | |
| model_type: bpe | |
| save_path: ./data/bpe_30 | |
| text_builder: | |
| func: src.tokenizer.gather_training_text | |
| manifest_path: ./data/mini_an4/manifest/train_nodev.tsv | |
| model: | |
| vocab_size: 30 | |
| token_list: ./data/bpe_30/tokens.txt | |
| encoder: vgg_rnn | |
| encoder_conf: | |
| num_layers: 1 | |
| hidden_size: 2 | |
| output_size: 2 | |
| decoder: rnn | |
| decoder_conf: | |
| hidden_size: 2 | |
| normalize: utterance_mvn | |
| normalize_conf: {} | |
| model_conf: | |
| ctc_weight: 0.3 | |
| lsm_weight: 0.1 | |
| length_normalized_loss: false | |
| frontend: default | |
| frontend_conf: | |
| n_fft: 512 | |
| win_length: 400 | |
| hop_length: 160 | |
| 2026-01-14 00:36:01 | INFO | espnet3 | Publish config content: | |
| pack_model: | |
| strategy: auto | |
| out_dir: exp/model_pack | |
| readme_template: egs3/TEMPLATE/asr/demo_templates/readme.md | |
| include: [] | |
| extra: [] | |
| exclude: | |
| - '**/*.log' | |
| - '**/tensorboard/**' | |
| - '**/wandb/**' | |
| files: | |
| asr_model_file: exp/train_asr_rnn_data_aug_debug/epoch0_step1_valid.acc.ckpt | |
| yaml_files: | |
| asr_train_config: exp/train_asr_rnn_data_aug_debug/config.yaml | |
| espnet2: | |
| task: asr | |
| yaml_files: {} | |
| files: {} | |
| option: [] | |
| upload_model: | |
| hf_repo: ms180/mini_an4_integration_test | |
| 2026-01-14 00:36:01 | INFO | espnet3 | === [START] stage: pack_model === | |