End of training

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+predictions_common_voice_13_en_common_voice_13_en_test_wer65.82_hyp.trn.snt.utt filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,122 @@
+---
+base_model: Lakoc/DeCRED_small_cv_2
+tags:
+- generated_from_trainer
+datasets:
+- common_voice_13_0
+metrics:
+- wer
+model-index:
+- name: DeCRED_small_cv_v2_linear_mixing
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+# DeCRED_small_cv_v2_linear_mixing
+
+This model is a fine-tuned version of [Lakoc/DeCRED_small_cv_2](https://huggingface.co/Lakoc/DeCRED_small_cv_2) on the common_voice_13_0 dataset.
+It achieves the following results on the evaluation set:
+- Loss: 1.9542
+- Cer: 0.3765
+- Wer: 0.6117
+- Mer: 0.5575
+- Wil: 0.7685
+- Wip: 0.2315
+- Hits: 22590
+- Substitutions: 20263
+- Deletions: 3668
+- Insertions: 4527
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 0.0005
+- train_batch_size: 128
+- eval_batch_size: 128
+- seed: 42
+- distributed_type: multi-GPU
+- num_devices: 2
+- gradient_accumulation_steps: 2
+- total_train_batch_size: 512
+- total_eval_batch_size: 256
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: linear
+- num_epochs: 50.0
+
+### Training results
+
+| Training Loss | Epoch | Step | Validation Loss | Cer     | Wer     | Mer    | Wil    | Wip    | Hits  | Substitutions | Deletions | Insertions |
+|:-------------:|:-----:|:----:|:---------------:|:-------:|:-------:|:------:|:------:|:------:|:-----:|:-------------:|:---------:|:----------:|
+| 6.9661        | 0.98  | 22   | 6.8477          | 60.1841 | 50.8357 | 0.9996 | 1.0000 | 0.0000 | 983   | 45533         | 5         | 2319391    |
+| 6.5874        | 2.0   | 45   | 6.6100          | 59.9264 | 50.2003 | 0.9995 | 1.0000 | 0.0000 | 1108  | 45404         | 9         | 2289956    |
+| 6.4843        | 2.98  | 67   | 6.3899          | 59.4350 | 49.5502 | 0.9995 | 1.0000 | 0.0000 | 1246  | 45261         | 14        | 2259851    |
+| 6.1871        | 4.0   | 90   | 6.1667          | 58.7677 | 48.7290 | 0.9994 | 1.0000 | 0.0000 | 1390  | 45122         | 9         | 2221790    |
+| 6.1088        | 4.98  | 112  | 5.9594          | 57.9917 | 47.9251 | 0.9993 | 1.0000 | 0.0000 | 1603  | 44900         | 18        | 2184606    |
+| 5.8041        | 6.0   | 135  | 5.7487          | 56.6337 | 46.6790 | 0.9992 | 1.0000 | 0.0000 | 1816  | 44680         | 25        | 2126851    |
+| 5.7494        | 6.98  | 157  | 5.5529          | 54.6535 | 44.8725 | 0.9991 | 1.0000 | 0.0000 | 1974  | 44513         | 34        | 2042966    |
+| 5.4083        | 8.0   | 180  | 5.3546          | 52.4198 | 42.9372 | 0.9989 | 0.9999 | 0.0001 | 2173  | 44298         | 50        | 1953135    |
+| 5.3779        | 8.98  | 202  | 5.1706          | 49.7925 | 40.7569 | 0.9988 | 0.9999 | 0.0001 | 2371  | 44091         | 59        | 1851904    |
+| 5.02          | 10.0  | 225  | 4.9842          | 46.7020 | 38.1816 | 0.9985 | 0.9999 | 0.0001 | 2603  | 43831         | 87        | 1732326    |
+| 4.9776        | 10.98 | 247  | 4.8119          | 43.6679 | 35.8707 | 0.9983 | 0.9999 | 0.0001 | 2852  | 43570         | 99        | 1625071    |
+| 4.7425        | 12.0  | 270  | 4.6377          | 39.7527 | 32.6943 | 0.9980 | 0.9999 | 0.0001 | 3054  | 43352         | 115       | 1477503    |
+| 4.608         | 12.98 | 292  | 4.4773          | 35.2066 | 29.0084 | 0.9976 | 0.9998 | 0.0002 | 3233  | 43132         | 156       | 1306210    |
+| 4.4031        | 14.0  | 315  | 4.3150          | 31.5887 | 26.0092 | 0.9971 | 0.9998 | 0.0002 | 3487  | 42835         | 199       | 1166942    |
+| 4.3239        | 14.98 | 337  | 4.1657          | 27.0209 | 22.5064 | 0.9965 | 0.9997 | 0.0003 | 3717  | 42481         | 323       | 1004215    |
+| 4.1256        | 16.0  | 360  | 4.0154          | 22.4586 | 18.8399 | 0.9956 | 0.9996 | 0.0004 | 3907  | 42238         | 376       | 833835     |
+| 4.0373        | 16.98 | 382  | 3.8773          | 18.2020 | 15.3318 | 0.9942 | 0.9995 | 0.0005 | 4128  | 41849         | 544       | 670857     |
+| 3.8293        | 18.0  | 405  | 3.7389          | 14.5637 | 12.3297 | 0.9923 | 0.9993 | 0.0007 | 4442  | 41435         | 644       | 531510     |
+| 3.7401        | 18.98 | 427  | 3.6120          | 11.4548 | 9.7572  | 0.9897 | 0.9990 | 0.0010 | 4708  | 41051         | 762       | 412101     |
+| 3.5255        | 20.0  | 450  | 3.4851          | 8.4210  | 7.3279  | 0.9852 | 0.9984 | 0.0016 | 5122  | 40427         | 972       | 299500     |
+| 3.5611        | 20.98 | 472  | 3.3694          | 5.8830  | 5.3130  | 0.9783 | 0.9974 | 0.0026 | 5473  | 39918         | 1130      | 206120     |
+| 3.3464        | 22.0  | 495  | 3.2537          | 4.1319  | 3.8709  | 0.9682 | 0.9959 | 0.0041 | 5905  | 39233         | 1383      | 139463     |
+| 3.3134        | 22.98 | 517  | 3.1489          | 3.1610  | 3.0400  | 0.9567 | 0.9940 | 0.0060 | 6408  | 38514         | 1599      | 101309     |
+| 3.1154        | 24.0  | 540  | 3.0447          | 2.2506  | 2.2882  | 0.9392 | 0.9909 | 0.0091 | 6887  | 37758         | 1876      | 66816      |
+| 3.0684        | 24.98 | 562  | 2.9503          | 1.5946  | 1.7552  | 0.9158 | 0.9861 | 0.0139 | 7503  | 36824         | 2194      | 42636      |
+| 2.9926        | 26.0  | 585  | 2.8569          | 1.2290  | 1.4535  | 0.8931 | 0.9808 | 0.0192 | 8097  | 36034         | 2390      | 29195      |
+| 2.9429        | 26.98 | 607  | 2.7728          | 1.0860  | 1.3147  | 0.8752 | 0.9757 | 0.0243 | 8722  | 35139         | 2660      | 23363      |
+| 2.8033        | 28.0  | 630  | 2.6900          | 0.8996  | 1.1624  | 0.8519 | 0.9687 | 0.0313 | 9399  | 34374         | 2748      | 16952      |
+| 2.7652        | 28.98 | 652  | 2.6158          | 0.8134  | 1.0854  | 0.8326 | 0.9615 | 0.0385 | 10155 | 33342         | 3024      | 14126      |
+| 2.6598        | 30.0  | 675  | 2.5430          | 0.7254  | 1.0033  | 0.8098 | 0.9526 | 0.0474 | 10964 | 32379         | 3178      | 11116      |
+| 2.6088        | 30.98 | 697  | 2.4781          | 0.6766  | 0.9584  | 0.7914 | 0.9439 | 0.0561 | 11755 | 31417         | 3349      | 9819       |
+| 2.5442        | 32.0  | 720  | 2.4151          | 0.6563  | 0.9343  | 0.7759 | 0.9354 | 0.0646 | 12556 | 30412         | 3553      | 9501       |
+| 2.5035        | 32.98 | 742  | 2.3592          | 0.6205  | 0.8964  | 0.7572 | 0.9252 | 0.0748 | 13370 | 29435         | 3716      | 8552       |
+| 2.4259        | 34.0  | 765  | 2.3051          | 0.5803  | 0.8567  | 0.7354 | 0.9123 | 0.0877 | 14341 | 28415         | 3765      | 7674       |
+| 2.3946        | 34.98 | 787  | 2.2576          | 0.5549  | 0.8295  | 0.7172 | 0.9004 | 0.0996 | 15216 | 27479         | 3826      | 7282       |
+| 2.3014        | 36.0  | 810  | 2.2121          | 0.5257  | 0.8003  | 0.6971 | 0.8864 | 0.1136 | 16180 | 26476         | 3865      | 6888       |
+| 2.2883        | 36.98 | 832  | 2.1725          | 0.5050  | 0.7753  | 0.6790 | 0.8733 | 0.1267 | 17049 | 25677         | 3795      | 6598       |
+| 2.2694        | 38.0  | 855  | 2.1350          | 0.4803  | 0.7461  | 0.6596 | 0.8587 | 0.1413 | 17913 | 24794         | 3814      | 6102       |
+| 2.2372        | 38.98 | 877  | 2.1028          | 0.4635  | 0.7254  | 0.6447 | 0.8465 | 0.1535 | 18597 | 24029         | 3895      | 5821       |
+| 2.1639        | 40.0  | 900  | 2.0728          | 0.4458  | 0.7033  | 0.6289 | 0.8335 | 0.1665 | 19309 | 23310         | 3902      | 5508       |
+| 2.1478        | 40.98 | 922  | 2.0475          | 0.4303  | 0.6843  | 0.6146 | 0.8211 | 0.1789 | 19960 | 22639         | 3922      | 5271       |
+| 2.1546        | 42.0  | 945  | 2.0245          | 0.4172  | 0.6653  | 0.5999 | 0.8083 | 0.1917 | 20644 | 22064         | 3813      | 5075       |
+| 2.1382        | 42.98 | 967  | 2.0056          | 0.4062  | 0.6510  | 0.5885 | 0.7979 | 0.2021 | 21179 | 21588         | 3754      | 4942       |
+| 2.1007        | 44.0  | 990  | 1.9892          | 0.3961  | 0.6376  | 0.5780 | 0.7881 | 0.2119 | 21656 | 21111         | 3754      | 4798       |
+| 2.09          | 44.98 | 1012 | 1.9766          | 0.3885  | 0.6282  | 0.5705 | 0.7810 | 0.2190 | 21996 | 20801         | 3724      | 4698       |
+| 2.1065        | 46.0  | 1035 | 1.9664          | 0.3827  | 0.6207  | 0.5644 | 0.7752 | 0.2248 | 22286 | 20556         | 3679      | 4641       |
+| 2.1115        | 46.98 | 1057 | 1.9596          | 0.3793  | 0.6157  | 0.5604 | 0.7713 | 0.2287 | 22466 | 20393         | 3662      | 4587       |
+| 2.0602        | 48.0  | 1080 | 1.9554          | 0.3770  | 0.6125  | 0.5581 | 0.7691 | 0.2309 | 22564 | 20295         | 3662      | 4537       |
+| 1.9657        | 48.89 | 1100 | 1.9542          | 0.3765  | 0.6117  | 0.5575 | 0.7685 | 0.2315 | 22590 | 20263         | 3668      | 4527       |
+
+
+### Framework versions
+
+- Transformers 4.40.0.dev0
+- Pytorch 2.2.0+rocm5.6
+- Datasets 2.18.0
+- Tokenizers 0.15.2

config.json

@@ -0,0 +1,276 @@
+{
+  "_name_or_path": "Lakoc/DeCRED_small_cv_2",
+  "architectures": [
+    "JointCTCAttentionEncoderDecoder"
+  ],
+  "bos_token_id": 0,
+  "ctc_weight": 0.3,
+  "decoder": {
+    "_name_or_path": "Lakoc/gpt2_256h_6l_add_head3_04",
+    "activation_function": "gelu_new",
+    "add_cross_attention": true,
+    "architectures": null,
+    "attn_pdrop": 0.1,
+    "average_logits": false,
+    "bad_words_ids": null,
+    "begin_suppress_tokens": null,
+    "bos_token_id": 0,
+    "chunk_size_feed_forward": 0,
+    "cross_attention_hidden_size": null,
+    "decoder_start_token_id": null,
+    "diversity_penalty": 0.0,
+    "do_sample": false,
+    "early_stopping": false,
+    "embd_pdrop": 0.1,
+    "encoder_no_repeat_ngram_size": 0,
+    "eos_token_id": 1,
+    "exponential_decay_length_penalty": null,
+    "finetuning_task": null,
+    "forced_bos_token_id": null,
+    "forced_eos_token_id": null,
+    "head_locations": [
+      3
+    ],
+    "head_weights": [
+      0.6,
+      0.4
+    ],
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "initializer_range": 0.02,
+    "is_decoder": true,
+    "is_encoder_decoder": false,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "layer_norm_epsilon": 1e-05,
+    "length_penalty": 1.0,
+    "max_length": 20,
+    "min_length": 0,
+    "model_type": "gpt2-multi-head",
+    "n_embd": 256,
+    "n_head": 4,
+    "n_inner": 2048,
+    "n_layer": 6,
+    "n_positions": 1024,
+    "no_repeat_ngram_size": 0,
+    "num_beam_groups": 1,
+    "num_beams": 1,
+    "num_return_sequences": 1,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "output_scores": false,
+    "pad_token_id": null,
+    "pos_emb_fixed": true,
+    "prefix": null,
+    "problem_type": null,
+    "pruned_heads": {},
+    "remove_invalid_values": false,
+    "reorder_and_upcast_attn": false,
+    "repetition_penalty": 1.0,
+    "resid_pdrop": 0.1,
+    "return_dict": true,
+    "return_dict_in_generate": false,
+    "scale_attn_by_inverse_layer_idx": false,
+    "scale_attn_weights": true,
+    "sep_token_id": null,
+    "summary_activation": null,
+    "summary_first_dropout": 0.1,
+    "summary_proj_to_labels": true,
+    "summary_type": "cls_index",
+    "summary_use_proj": true,
+    "suppress_tokens": null,
+    "task_specific_params": null,
+    "temperature": 1.0,
+    "tf_legacy_loss": false,
+    "tie_additional_weights": false,
+    "tie_encoder_decoder": false,
+    "tie_word_embeddings": false,
+    "tokenizer_class": null,
+    "top_k": 50,
+    "top_p": 1.0,
+    "torch_dtype": null,
+    "torchscript": false,
+    "typical_p": 1.0,
+    "use_bfloat16": false,
+    "use_cache": true,
+    "vocab_size": 1000
+  },
+  "decoder_start_token_id": 0,
+  "encoder": {
+    "_name_or_path": "Lakoc/fisher_ebranchformer_enc_12_layers_fixed",
+    "activation_dropout": 0.1,
+    "adapter_attn_dim": null,
+    "adapter_kernel_size": 3,
+    "adapter_stride": 2,
+    "add_adapter": false,
+    "add_cross_attention": false,
+    "apply_spec_augment": false,
+    "architectures": null,
+    "attention_dropout": 0.1,
+    "bad_words_ids": null,
+    "begin_suppress_tokens": null,
+    "bos_token_id": 1,
+    "chunk_size_feed_forward": 0,
+    "classifier_proj_size": 256,
+    "codevector_dim": 256,
+    "conformer_conv_dropout": 0.1,
+    "contrastive_logits_temperature": 0.1,
+    "conv_bias": false,
+    "conv_depthwise_kernel_size": 31,
+    "conv_dim": [
+      256,
+      256
+    ],
+    "conv_kernel": [
+      3,
+      3
+    ],
+    "conv_padding": [
+      1,
+      1
+    ],
+    "conv_stride": [
+      2,
+      2
+    ],
+    "cross_attention_hidden_size": null,
+    "csgu_activation": "identity",
+    "csgu_conv_dropout": 0.1,
+    "csgu_kernel_size": 31,
+    "csgu_use_linear_after_conv": false,
+    "ctc_loss_reduction": "mean",
+    "ctc_zero_infinity": true,
+    "decoder_start_token_id": null,
+    "diversity_loss_weight": 0.1,
+    "diversity_penalty": 0.0,
+    "do_sample": false,
+    "do_stable_layer_norm": false,
+    "early_stopping": false,
+    "encoder_no_repeat_ngram_size": 0,
+    "eos_token_id": 2,
+    "exponential_decay_length_penalty": null,
+    "fe_position_embeddings": false,
+    "feat_extract_activation": "gelu",
+    "feat_extract_norm": "group",
+    "feat_proj_dropout": 0.0,
+    "feat_quantizer_dropout": 0.0,
+    "final_dropout": 0.1,
+    "finetuning_task": null,
+    "forced_bos_token_id": null,
+    "forced_eos_token_id": null,
+    "hidden_act": "gelu",
+    "hidden_dropout": 0.1,
+    "hidden_size": 256,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "initializer_range": 0.02,
+    "intermediate_size": 1024,
+    "is_causal": false,
+    "is_decoder": false,
+    "is_encoder_decoder": false,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "layer_norm_eps": 1e-05,
+    "layerdrop": 0.0,
+    "length_penalty": 1.0,
+    "mask_feature_length": 10,
+    "mask_feature_min_masks": 0,
+    "mask_feature_prob": 0.0,
+    "mask_time_length": 10,
+    "mask_time_min_masks": 2,
+    "mask_time_prob": 0.05,
+    "max_length": 20,
+    "max_source_positions": 1024,
+    "merge_conv_kernel": 31,
+    "min_length": 0,
+    "model_type": "wav2vec2-ebranchformer",
+    "no_repeat_ngram_size": 0,
+    "num_adapter_layers": 3,
+    "num_attention_heads": 4,
+    "num_beam_groups": 1,
+    "num_beams": 1,
+    "num_codevector_groups": 2,
+    "num_codevectors_per_group": 320,
+    "num_conv_pos_embedding_groups": 16,
+    "num_conv_pos_embeddings": 128,
+    "num_fbanks": 80,
+    "num_feat_extract_layers": 2,
+    "num_hidden_layers": 12,
+    "num_mel_bins": 80,
+    "num_negatives": 100,
+    "num_return_sequences": 1,
+    "output_attentions": false,
+    "output_hidden_size": 256,
+    "output_hidden_states": false,
+    "output_scores": false,
+    "pad_token_id": 3,
+    "position_embeddings_type": "relative",
+    "prefix": null,
+    "problem_type": null,
+    "proj_codevector_dim": 256,
+    "pruned_heads": {},
+    "remove_invalid_values": false,
+    "repetition_penalty": 1.0,
+    "return_dict": true,
+    "return_dict_in_generate": false,
+    "rotary_embedding_base": 10000,
+    "sep_token_id": null,
+    "suppress_tokens": null,
+    "task_specific_params": null,
+    "tdnn_dilation": [
+      1,
+      2,
+      3,
+      1,
+      1
+    ],
+    "tdnn_dim": [
+      512,
+      512,
+      512,
+      512,
+      1500
+    ],
+    "tdnn_kernel": [
+      5,
+      3,
+      3,
+      1,
+      1
+    ],
+    "temperature": 1.0,
+    "tf_legacy_loss": false,
+    "tie_encoder_decoder": false,
+    "tie_word_embeddings": true,
+    "tokenizer_class": null,
+    "top_k": 50,
+    "top_p": 1.0,
+    "torch_dtype": null,
+    "torchscript": false,
+    "typical_p": 1.0,
+    "use_bfloat16": false,
+    "use_fbanks": true,
+    "use_macaron_ff": true,
+    "use_weighted_layer_sum": false,
+    "vocab_size": 1000,
+    "xvector_output_dim": 512
+  },
+  "eos_token_id": 1,
+  "is_encoder_decoder": true,
+  "lsm_factor": 0.1,
+  "mask_token_id": 4,
+  "model_type": "joint_aed_ctc_speech-encoder-decoder",
+  "pad_token_id": 3,
+  "shared_lm_head": false,
+  "tie_word_embeddings": false,
+  "torch_dtype": "float32",
+  "transformers_version": "4.40.0.dev0"
+}

generation_config.json

@@ -0,0 +1,15 @@
+{
+  "apply_eos_space_trick": false,
+  "bos_token_id": 0,
+  "ctc_margin": 0,
+  "ctc_weight": 0.0,
+  "decoder_start_token_id": 0,
+  "eos_space_trick_weight": 0.0,
+  "eos_token_id": 1,
+  "lm_model": null,
+  "lm_weight": 0.0,
+  "max_length": 512,
+  "pad_token_id": 3,
+  "space_token_id": -1,
+  "transformers_version": "4.40.0.dev0"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b1f65da0590cfca84160eba8192738015ae58b385b9437b481dbafcf918691df
+size 144251296

predictions_common_voice_13_en_common_voice_13_en_test_wer65.82_hyp.trn.snt.utt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8c3cc8bc6c5185c449f55bec70b2806aaacc3fad228628dcdfd02931fc06704a
+size 11916777

predictions_common_voice_13_en_common_voice_13_en_test_wer65.82_hyp.trn.sys

@@ -0,0 +1,18 @@
+
+
+
+                     SYSTEM SUMMARY PERCENTAGES by SPEAKER                      
+
+,------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------.
+|/scratch/project_465000836/ipoloka/huggingface_asr/experiments/decred/commonvoice/DeCRED_small_cv_v2_linear_mixing/predictions_common_voice_13_en_common_voice_13_en_test_wer65.82_hyp.trn|
+|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+|       SPKR          |        # Snt                 # Wrd         |       Corr                 Sub                 Del                 Ins                 Err               S.Err        |
+|---------------------+--------------------------------------------+-----------------------------------------------------------------------------------------------------------------------|
+|       utt           |       16384                 144172         |       45.3                45.5                 9.2                11.1                65.8                99.7        |
+|==========================================================================================================================================================================================|
+|       Sum/Avg       |       16384                 144172         |       45.3                45.5                 9.2                11.1                65.8                99.7        |
+|==========================================================================================================================================================================================|
+|        Mean         |      16384.0               144172.0        |       45.3                45.5                 9.2                11.1                65.8                99.7        |
+|        S.D.         |         0.0                   0.0          |        0.0                 0.0                 0.0                 0.0                 0.0                 0.0        |
+|       Median        |      16384.0               144172.0        |       45.3                45.5                 9.2                11.1                65.8                99.7        |
+`------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------'

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0ff497cfcbd06e8b89e33e643f15a1d2ece1546262835fc12512c930934d82b4
+size 5752