End of training

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+predictions_common_voice_13_en_common_voice_13_en_test_wer19.46_hyp.trn.snt.utt filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,83 @@
+---
+base_model: Lakoc/DeCRED_small_cv_2
+tags:
+- generated_from_trainer
+datasets:
+- common_voice_13_0
+metrics:
+- wer
+model-index:
+- name: DeCRED_linear_mixing_tuning
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+# DeCRED_linear_mixing_tuning
+
+This model is a fine-tuned version of [Lakoc/DeCRED_small_cv_2](https://huggingface.co/Lakoc/DeCRED_small_cv_2) on the common_voice_13_0 dataset.
+It achieves the following results on the evaluation set:
+- Loss: 1.0590
+- Cer: 0.0632
+- Wer: 0.1471
+- Mer: 0.1444
+- Wil: 0.2408
+- Wip: 0.7592
+- Hits: 23158
+- Substitutions: 2931
+- Deletions: 484
+- Insertions: 494
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 0.05
+- train_batch_size: 256
+- eval_batch_size: 64
+- seed: 42
+- distributed_type: multi-GPU
+- gradient_accumulation_steps: 4
+- total_train_batch_size: 1024
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: linear
+- num_epochs: 50.0
+
+### Training results
+
+| Training Loss | Epoch | Step | Validation Loss | Cer    | Wer    | Mer    | Wil    | Wip    | Hits  | Substitutions | Deletions | Insertions |
+|:-------------:|:-----:|:----:|:---------------:|:------:|:------:|:------:|:------:|:------:|:-----:|:-------------:|:---------:|:----------:|
+| 1.185         | 2.67  | 20   | 1.1215          | 0.0646 | 0.1520 | 0.1492 | 0.2479 | 0.7521 | 23030 | 3013          | 530       | 496        |
+| 1.0639        | 5.33  | 40   | 1.0717          | 0.0634 | 0.1480 | 0.1453 | 0.2420 | 0.7580 | 23124 | 2939          | 510       | 483        |
+| 1.0957        | 8.0   | 60   | 1.0612          | 0.0628 | 0.1464 | 0.1438 | 0.2402 | 0.7598 | 23162 | 2929          | 482       | 480        |
+| 1.0936        | 10.67 | 80   | 1.0595          | 0.0631 | 0.1469 | 0.1442 | 0.2407 | 0.7593 | 23158 | 2934          | 481       | 488        |
+| 1.0804        | 13.33 | 100  | 1.0591          | 0.0630 | 0.1468 | 0.1441 | 0.2405 | 0.7595 | 23164 | 2929          | 480       | 492        |
+| 1.1044        | 16.0  | 120  | 1.0591          | 0.0631 | 0.1469 | 0.1442 | 0.2405 | 0.7595 | 23162 | 2929          | 482       | 492        |
+| 1.0836        | 18.67 | 140  | 1.0589          | 0.0631 | 0.1468 | 0.1442 | 0.2405 | 0.7595 | 23163 | 2929          | 481       | 492        |
+| 1.0924        | 21.33 | 160  | 1.0590          | 0.0632 | 0.1471 | 0.1444 | 0.2408 | 0.7592 | 23158 | 2931          | 484       | 494        |
+| 1.1048        | 24.0  | 180  | 1.0590          | 0.0632 | 0.1471 | 0.1444 | 0.2408 | 0.7592 | 23158 | 2931          | 484       | 494        |
+| 1.0858        | 26.67 | 200  | 1.0589          | 0.0631 | 0.1470 | 0.1443 | 0.2407 | 0.7593 | 23160 | 2929          | 484       | 494        |
+| 1.0953        | 29.33 | 220  | 1.0589          | 0.0631 | 0.1468 | 0.1442 | 0.2405 | 0.7595 | 23163 | 2929          | 481       | 492        |
+| 1.1308        | 32.0  | 240  | 1.0590          | 0.0632 | 0.1471 | 0.1444 | 0.2408 | 0.7592 | 23158 | 2931          | 484       | 494        |
+
+
+### Framework versions
+
+- Transformers 4.40.0.dev0
+- Pytorch 2.2.0+rocm5.6
+- Datasets 2.18.0
+- Tokenizers 0.15.2

config.json

@@ -0,0 +1,276 @@
+{
+  "_name_or_path": "Lakoc/DeCRED_small_cv_2",
+  "architectures": [
+    "JointCTCAttentionEncoderDecoder"
+  ],
+  "bos_token_id": 0,
+  "ctc_weight": 0.3,
+  "decoder": {
+    "_name_or_path": "Lakoc/gpt2_256h_6l_add_head3_04",
+    "activation_function": "gelu_new",
+    "add_cross_attention": true,
+    "architectures": null,
+    "attn_pdrop": 0.1,
+    "average_logits": false,
+    "bad_words_ids": null,
+    "begin_suppress_tokens": null,
+    "bos_token_id": 0,
+    "chunk_size_feed_forward": 0,
+    "cross_attention_hidden_size": null,
+    "decoder_start_token_id": null,
+    "diversity_penalty": 0.0,
+    "do_sample": false,
+    "early_stopping": false,
+    "embd_pdrop": 0.1,
+    "encoder_no_repeat_ngram_size": 0,
+    "eos_token_id": 1,
+    "exponential_decay_length_penalty": null,
+    "finetuning_task": null,
+    "forced_bos_token_id": null,
+    "forced_eos_token_id": null,
+    "head_locations": [
+      3
+    ],
+    "head_weights": [
+      0.6,
+      0.4
+    ],
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "initializer_range": 0.02,
+    "is_decoder": true,
+    "is_encoder_decoder": false,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "layer_norm_epsilon": 1e-05,
+    "length_penalty": 1.0,
+    "max_length": 20,
+    "min_length": 0,
+    "model_type": "gpt2-multi-head",
+    "n_embd": 256,
+    "n_head": 4,
+    "n_inner": 2048,
+    "n_layer": 6,
+    "n_positions": 1024,
+    "no_repeat_ngram_size": 0,
+    "num_beam_groups": 1,
+    "num_beams": 1,
+    "num_return_sequences": 1,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "output_scores": false,
+    "pad_token_id": null,
+    "pos_emb_fixed": true,
+    "prefix": null,
+    "problem_type": null,
+    "pruned_heads": {},
+    "remove_invalid_values": false,
+    "reorder_and_upcast_attn": false,
+    "repetition_penalty": 1.0,
+    "resid_pdrop": 0.1,
+    "return_dict": true,
+    "return_dict_in_generate": false,
+    "scale_attn_by_inverse_layer_idx": false,
+    "scale_attn_weights": true,
+    "sep_token_id": null,
+    "summary_activation": null,
+    "summary_first_dropout": 0.1,
+    "summary_proj_to_labels": true,
+    "summary_type": "cls_index",
+    "summary_use_proj": true,
+    "suppress_tokens": null,
+    "task_specific_params": null,
+    "temperature": 1.0,
+    "tf_legacy_loss": false,
+    "tie_additional_weights": false,
+    "tie_encoder_decoder": false,
+    "tie_word_embeddings": false,
+    "tokenizer_class": null,
+    "top_k": 50,
+    "top_p": 1.0,
+    "torch_dtype": null,
+    "torchscript": false,
+    "typical_p": 1.0,
+    "use_bfloat16": false,
+    "use_cache": true,
+    "vocab_size": 1000
+  },
+  "decoder_start_token_id": 0,
+  "encoder": {
+    "_name_or_path": "Lakoc/fisher_ebranchformer_enc_12_layers_fixed",
+    "activation_dropout": 0.1,
+    "adapter_attn_dim": null,
+    "adapter_kernel_size": 3,
+    "adapter_stride": 2,
+    "add_adapter": false,
+    "add_cross_attention": false,
+    "apply_spec_augment": false,
+    "architectures": null,
+    "attention_dropout": 0.1,
+    "bad_words_ids": null,
+    "begin_suppress_tokens": null,
+    "bos_token_id": 1,
+    "chunk_size_feed_forward": 0,
+    "classifier_proj_size": 256,
+    "codevector_dim": 256,
+    "conformer_conv_dropout": 0.1,
+    "contrastive_logits_temperature": 0.1,
+    "conv_bias": false,
+    "conv_depthwise_kernel_size": 31,
+    "conv_dim": [
+      256,
+      256
+    ],
+    "conv_kernel": [
+      3,
+      3
+    ],
+    "conv_padding": [
+      1,
+      1
+    ],
+    "conv_stride": [
+      2,
+      2
+    ],
+    "cross_attention_hidden_size": null,
+    "csgu_activation": "identity",
+    "csgu_conv_dropout": 0.1,
+    "csgu_kernel_size": 31,
+    "csgu_use_linear_after_conv": false,
+    "ctc_loss_reduction": "mean",
+    "ctc_zero_infinity": true,
+    "decoder_start_token_id": null,
+    "diversity_loss_weight": 0.1,
+    "diversity_penalty": 0.0,
+    "do_sample": false,
+    "do_stable_layer_norm": false,
+    "early_stopping": false,
+    "encoder_no_repeat_ngram_size": 0,
+    "eos_token_id": 2,
+    "exponential_decay_length_penalty": null,
+    "fe_position_embeddings": false,
+    "feat_extract_activation": "gelu",
+    "feat_extract_norm": "group",
+    "feat_proj_dropout": 0.0,
+    "feat_quantizer_dropout": 0.0,
+    "final_dropout": 0.1,
+    "finetuning_task": null,
+    "forced_bos_token_id": null,
+    "forced_eos_token_id": null,
+    "hidden_act": "gelu",
+    "hidden_dropout": 0.1,
+    "hidden_size": 256,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "initializer_range": 0.02,
+    "intermediate_size": 1024,
+    "is_causal": false,
+    "is_decoder": false,
+    "is_encoder_decoder": false,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "layer_norm_eps": 1e-05,
+    "layerdrop": 0.0,
+    "length_penalty": 1.0,
+    "mask_feature_length": 10,
+    "mask_feature_min_masks": 0,
+    "mask_feature_prob": 0.0,
+    "mask_time_length": 10,
+    "mask_time_min_masks": 2,
+    "mask_time_prob": 0.05,
+    "max_length": 20,
+    "max_source_positions": 1024,
+    "merge_conv_kernel": 31,
+    "min_length": 0,
+    "model_type": "wav2vec2-ebranchformer",
+    "no_repeat_ngram_size": 0,
+    "num_adapter_layers": 3,
+    "num_attention_heads": 4,
+    "num_beam_groups": 1,
+    "num_beams": 1,
+    "num_codevector_groups": 2,
+    "num_codevectors_per_group": 320,
+    "num_conv_pos_embedding_groups": 16,
+    "num_conv_pos_embeddings": 128,
+    "num_fbanks": 80,
+    "num_feat_extract_layers": 2,
+    "num_hidden_layers": 12,
+    "num_mel_bins": 80,
+    "num_negatives": 100,
+    "num_return_sequences": 1,
+    "output_attentions": false,
+    "output_hidden_size": 256,
+    "output_hidden_states": false,
+    "output_scores": false,
+    "pad_token_id": 3,
+    "position_embeddings_type": "relative",
+    "prefix": null,
+    "problem_type": null,
+    "proj_codevector_dim": 256,
+    "pruned_heads": {},
+    "remove_invalid_values": false,
+    "repetition_penalty": 1.0,
+    "return_dict": true,
+    "return_dict_in_generate": false,
+    "rotary_embedding_base": 10000,
+    "sep_token_id": null,
+    "suppress_tokens": null,
+    "task_specific_params": null,
+    "tdnn_dilation": [
+      1,
+      2,
+      3,
+      1,
+      1
+    ],
+    "tdnn_dim": [
+      512,
+      512,
+      512,
+      512,
+      1500
+    ],
+    "tdnn_kernel": [
+      5,
+      3,
+      3,
+      1,
+      1
+    ],
+    "temperature": 1.0,
+    "tf_legacy_loss": false,
+    "tie_encoder_decoder": false,
+    "tie_word_embeddings": true,
+    "tokenizer_class": null,
+    "top_k": 50,
+    "top_p": 1.0,
+    "torch_dtype": null,
+    "torchscript": false,
+    "typical_p": 1.0,
+    "use_bfloat16": false,
+    "use_fbanks": true,
+    "use_macaron_ff": true,
+    "use_weighted_layer_sum": false,
+    "vocab_size": 1000,
+    "xvector_output_dim": 512
+  },
+  "eos_token_id": 1,
+  "is_encoder_decoder": true,
+  "lsm_factor": 0.1,
+  "mask_token_id": 4,
+  "model_type": "joint_aed_ctc_speech-encoder-decoder",
+  "pad_token_id": 3,
+  "shared_lm_head": false,
+  "tie_word_embeddings": false,
+  "torch_dtype": "float32",
+  "transformers_version": "4.40.0.dev0"
+}

generation_config.json

@@ -0,0 +1,15 @@
+{
+  "apply_eos_space_trick": false,
+  "bos_token_id": 0,
+  "ctc_margin": 0,
+  "ctc_weight": 0.0,
+  "decoder_start_token_id": 0,
+  "eos_space_trick_weight": 0.0,
+  "eos_token_id": 1,
+  "lm_model": null,
+  "lm_weight": 0.0,
+  "max_length": 512,
+  "pad_token_id": 3,
+  "space_token_id": -1,
+  "transformers_version": "4.40.0.dev0"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2d38c60c905689cd2eb72d6225b6037af9e159ba63b6684421cdc867bd21ce74
+size 144243304

predictions_common_voice_13_en_common_voice_13_en_test_wer19.46_hyp.trn.snt.utt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f90be3dde6e2a07d762777ce7503e51592a2400ef55baa2c15c3e70b36d12798
+size 11431046

predictions_common_voice_13_en_common_voice_13_en_test_wer19.46_hyp.trn.sys

@@ -0,0 +1,18 @@
+
+
+
+                     SYSTEM SUMMARY PERCENTAGES by SPEAKER                      
+
+,-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------.
+|/scratch/project_465000836/ipoloka/huggingface_asr/experiments/decred/commonvoice/DeCRED_linear_mixing_tuning/predictions_common_voice_13_en_common_voice_13_en_test_wer19.46_hyp.trn|
+|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+|       SPKR         |        # Snt                 # Wrd        |       Corr                 Sub                Del                 Ins                Err               S.Err       |
+|--------------------+-------------------------------------------+--------------------------------------------------------------------------------------------------------------------|
+|       utt          |       16365                 144023        |       83.2                14.2                2.6                 2.7               19.5                63.1       |
+|=====================================================================================================================================================================================|
+|       Sum/Avg      |       16365                 144023        |       83.2                14.2                2.6                 2.7               19.5                63.1       |
+|=====================================================================================================================================================================================|
+|        Mean        |      16365.0               144023.0       |       83.2                14.2                2.6                 2.7               19.5                63.1       |
+|        S.D.        |         0.0                   0.0         |        0.0                 0.0                0.0                 0.0                0.0                 0.0       |
+|       Median       |      16365.0               144023.0       |       83.2                14.2                2.6                 2.7               19.5                63.1       |
+`-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------'

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8d149e7d20e1596cfcc0fbf07335f4e8275568f7a342982420a9e80721de1adb
+size 5688