docs/source/fst-based-forced-alignment/kaldi-based.rst

Kaldi-based forced alignment
============================

This section describes in detail how to use `kaldi-decoder`_
for **FST-based** ``forced alignment`` with models trained by `CTC`_ loss.

.. hint::

  We have a colab notebook walking you through this section step by step.

  |kaldi-based forced alignment colab notebook|

  .. |kaldi-based forced alignment colab notebook| image:: https://colab.research.google.com/assets/colab-badge.svg
     :target: https://github.com/k2-fsa/colab/blob/master/icefall/ctc_forced_alignment_fst_based_kaldi.ipynb

Prepare the environment
-----------------------

Before you continue, make sure you have setup `icefall`_ by following :ref:`install icefall`.

.. hint::

   You don't need to install `Kaldi`_. We will ``NOT`` use `Kaldi`_ below.

Get the test data
-----------------

We use the test wave
from `CTC FORCED ALIGNMENT API TUTORIAL <https://pytorch.org/audio/main/tutorials/ctc_forced_alignment_api_tutorial.html>`_

.. code-block:: python3

  import torchaudio

  # Download test wave
  speech_file = torchaudio.utils.download_asset("tutorial-assets/Lab41-SRI-VOiCES-src-sp0307-ch127535-sg0042.wav")
  print(speech_file)
  waveform, sr = torchaudio.load(speech_file)
  transcript = "i had that curiosity beside me at this moment".split()
  print(waveform.shape, sr)

  assert waveform.ndim == 2
  assert waveform.shape[0] == 1
  assert sr == 16000

The test wave is downloaded to::

  $HOME/.cache/torch/hub/torchaudio/tutorial-assets/Lab41-SRI-VOiCES-src-sp0307-ch127535-sg0042.wav

.. raw:: html

  <table>
    <tr>
      <th>Wave filename</th>
      <th>Content</th>
      <th>Text</th>
    </tr>
    <tr>
      <td>Lab41-SRI-VOiCES-src-sp0307-ch127535-sg0042.wav</td>
      <td>
       <audio title="Lab41-SRI-VOiCES-src-sp0307-ch127535-sg0042.wav" controls="controls">
             <source src="/icefall/_static/kaldi-align/Lab41-SRI-VOiCES-src-sp0307-ch127535-sg0042.wav" type="audio/wav">
             Your browser does not support the <code>audio</code> element.
       </audio>
      </td>
      <td>
        i had that curiosity beside me at this moment
      </td>
    </tr>
  </table>

We use the test model
from `CTC FORCED ALIGNMENT API TUTORIAL <https://pytorch.org/audio/main/tutorials/ctc_forced_alignment_api_tutorial.html>`_

.. code-block:: python3

  import torch

  bundle = torchaudio.pipelines.MMS_FA

  device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
  model = bundle.get_model(with_star=False).to(device)

The model is downloaded to::

  $HOME/.cache/torch/hub/checkpoints/model.pt

Compute log_probs
-----------------

.. code-block:: bash

  with torch.inference_mode():
      emission, _ = model(waveform.to(device))
      print(emission.shape)

It should print::

  torch.Size([1, 169, 28])

Create token2id and id2token
----------------------------

.. code-block:: python3

    token2id = bundle.get_dict(star=None)
    id2token = {i:t for t, i in token2id.items()}
    token2id["<eps>"] = 0
    del token2id["-"]

Create word2id and id2word
--------------------------

.. code-block:: python3

  words = list(set(transcript))
  word2id = dict()
  word2id['eps'] = 0
  for i, w in enumerate(words):
    word2id[w] = i + 1

  id2word = {i:w for w, i in word2id.items()}

Note that we only use words from the transcript of the test wave.

Generate lexicon-related files
------------------------------

We use the code below to generate the following 4 files:

  - ``lexicon.txt``
  - ``tokens.txt``
  - ``words.txt``
  - ``lexicon_disambig.txt``

.. caution::

   ``words.txt`` contains only words from the transcript of the test wave.

.. code-block:: python3

  from prepare_lang import add_disambig_symbols

  lexicon = [(w, list(w)) for w in word2id if w != "eps"]
  lexicon_disambig, max_disambig_id = add_disambig_symbols(lexicon)

  with open('lexicon.txt', 'w', encoding='utf-8') as f:
    for w, tokens in lexicon:
      f.write(f"{w} {' '.join(tokens)}\n")

  with open('lexicon_disambig.txt', 'w', encoding='utf-8') as f:
    for w, tokens in lexicon_disambig:
      f.write(f"{w} {' '.join(tokens)}\n")

  with open('tokens.txt', 'w', encoding='utf-8') as f:
    for t, i in token2id.items():
      if t == '-':
        t = "<eps>"
      f.write(f"{t} {i}\n")

    for k in range(max_disambig_id + 2):
      f.write(f"#{k} {len(token2id) + k}\n")

  with open('words.txt', 'w', encoding='utf-8') as f:
    for w, i in word2id.items():
      f.write(f"{w} {i}\n")
    f.write(f'#0 {len(word2id)}\n')


To give you an idea about what the generated files look like::

  head -n 50 lexicon.txt lexicon_disambig.txt tokens.txt words.txt

prints::

  ==> lexicon.txt <==
  moment m o m e n t
  beside b e s i d e
  i i
  this t h i s
  curiosity c u r i o s i t y
  had h a d
  that t h a t
  at a t
  me m e

  ==> lexicon_disambig.txt <==
  moment m o m e n t
  beside b e s i d e
  i i
  this t h i s
  curiosity c u r i o s i t y
  had h a d
  that t h a t
  at a t
  me m e

  ==> tokens.txt <==
  a 1
  i 2
  e 3
  n 4
  o 5
  u 6
  t 7
  s 8
  r 9
  m 10
  k 11
  l 12
  d 13
  g 14
  h 15
  y 16
  b 17
  p 18
  w 19
  c 20
  v 21
  j 22
  z 23
  f 24
  ' 25
  q 26
  x 27
  <eps> 0
  #0 28
  #1 29

  ==> words.txt <==
  eps 0
  moment 1
  beside 2
  i 3
  this 4
  curiosity 5
  had 6
  that 7
  at 8
  me 9
  #0 10

.. note::

   This test model uses characters as modeling unit. If you use other types of
   modeling unit, the same code can be used without any change.

Convert transcript to an FST graph
----------------------------------

.. code-block:: bash

   egs/librispeech/ASR/local/prepare_lang_fst.py --lang-dir ./

The above command should generate two files ``H.fst`` and ``HL.fst``. We will
use ``HL.fst`` below::

  -rw-r--r-- 1 root root  13K Jun 12 08:28 H.fst
  -rw-r--r-- 1 root root 3.7K Jun 12 08:28 HL.fst

Force aligner
-------------

Now, everything is ready. We can use the following code to get forced alignments.

.. code-block:: python3

  from kaldi_decoder import DecodableCtc, FasterDecoder, FasterDecoderOptions
  import kaldifst

  def force_align():
      HL = kaldifst.StdVectorFst.read("./HL.fst")
      decodable = DecodableCtc(emission[0].contiguous().cpu().numpy())
      decoder_opts = FasterDecoderOptions(max_active=3000)
      decoder = FasterDecoder(HL, decoder_opts)
      decoder.decode(decodable)
      if not decoder.reached_final():
          print(f"failed to decode xxx")
          return None
      ok, best_path = decoder.get_best_path()

      (
          ok,
          isymbols_out,
          osymbols_out,
          total_weight,
      ) = kaldifst.get_linear_symbol_sequence(best_path)
      if not ok:
          print(f"failed to get linear symbol sequence for xxx")
          return None

      # We need to use i-1 here since we have incremented tokens during
      # HL construction
      alignment = [i-1 for i in isymbols_out]
      return alignment

  alignment = force_align()

  for i, a in enumerate(alignment):
    print(i, id2token[a])

The output should be identical to
`<https://pytorch.org/audio/main/tutorials/ctc_forced_alignment_api_tutorial.html#frame-level-alignments>`_.

For ease of reference, we list the output below::

  0 -
  1 -
  2 -
  3 -
  4 -
  5 -
  6 -
  7 -
  8 -
  9 -
  10 -
  11 -
  12 -
  13 -
  14 -
  15 -
  16 -
  17 -
  18 -
  19 -
  20 -
  21 -
  22 -
  23 -
  24 -
  25 -
  26 -
  27 -
  28 -
  29 -
  30 -
  31 -
  32 i
  33 -
  34 -
  35 h
  36 h
  37 a
  38 -
  39 -
  40 -
  41 d
  42 -
  43 -
  44 t
  45 h
  46 -
  47 a
  48 -
  49 -
  50 t
  51 -
  52 -
  53 -
  54 c
  55 -
  56 -
  57 -
  58 u
  59 u
  60 -
  61 -
  62 -
  63 r
  64 -
  65 i
  66 -
  67 -
  68 -
  69 -
  70 -
  71 -
  72 o
  73 -
  74 -
  75 -
  76 -
  77 -
  78 -
  79 s
  80 -
  81 -
  82 -
  83 i
  84 -
  85 t
  86 -
  87 -
  88 y
  89 -
  90 -
  91 -
  92 -
  93 b
  94 -
  95 e
  96 -
  97 -
  98 -
  99 -
  100 -
  101 s
  102 -
  103 -
  104 -
  105 -
  106 -
  107 -
  108 -
  109 -
  110 i
  111 -
  112 -
  113 d
  114 e
  115 -
  116 m
  117 -
  118 -
  119 e
  120 -
  121 -
  122 -
  123 -
  124 a
  125 -
  126 -
  127 t
  128 -
  129 t
  130 h
  131 -
  132 i
  133 -
  134 -
  135 -
  136 s
  137 -
  138 -
  139 -
  140 -
  141 m
  142 -
  143 -
  144 o
  145 -
  146 -
  147 -
  148 m
  149 -
  150 -
  151 e
  152 -
  153 n
  154 -
  155 t
  156 -
  157 -
  158 -
  159 -
  160 -
  161 -
  162 -
  163 -
  164 -
  165 -
  166 -
  167 -
  168 -

To merge tokens, we use::

  from icefall.ctc import merge_tokens
  token_spans = merge_tokens(alignment)
  for span in token_spans:
    print(id2token[span.token], span.start, span.end)

The output is given below::

  i 32 33
  h 35 37
  a 37 38
  d 41 42
  t 44 45
  h 45 46
  a 47 48
  t 50 51
  c 54 55
  u 58 60
  r 63 64
  i 65 66
  o 72 73
  s 79 80
  i 83 84
  t 85 86
  y 88 89
  b 93 94
  e 95 96
  s 101 102
  i 110 111
  d 113 114
  e 114 115
  m 116 117
  e 119 120
  a 124 125
  t 127 128
  t 129 130
  h 130 131
  i 132 133
  s 136 137
  m 141 142
  o 144 145
  m 148 149
  e 151 152
  n 153 154
  t 155 156

All of the code below is copied and modified
from `<https://pytorch.org/audio/main/tutorials/ctc_forced_alignment_api_tutorial.html>`_.

Segment each word using the computed alignments
-----------------------------------------------

.. code-block:: python3

  def unflatten(list_, lengths):
      assert len(list_) == sum(lengths)
      i = 0
      ret = []
      for l in lengths:
          ret.append(list_[i : i + l])
          i += l
      return ret


  word_spans = unflatten(token_spans, [len(word) for word in transcript])
  print(word_spans)

The output is::

  [[TokenSpan(token=2, start=32, end=33)],
   [TokenSpan(token=15, start=35, end=37), TokenSpan(token=1, start=37, end=38), TokenSpan(token=13, start=41, end=42)],
   [TokenSpan(token=7, start=44, end=45), TokenSpan(token=15, start=45, end=46), TokenSpan(token=1, start=47, end=48), TokenSpan(token=7, start=50, end=51)],
   [TokenSpan(token=20, start=54, end=55), TokenSpan(token=6, start=58, end=60), TokenSpan(token=9, start=63, end=64), TokenSpan(token=2, start=65, end=66), TokenSpan(token=5, start=72, end=73), TokenSpan(token=8, start=79, end=80), TokenSpan(token=2, start=83, end=84), TokenSpan(token=7, start=85, end=86), TokenSpan(token=16, start=88, end=89)],
   [TokenSpan(token=17, start=93, end=94), TokenSpan(token=3, start=95, end=96), TokenSpan(token=8, start=101, end=102), TokenSpan(token=2, start=110, end=111), TokenSpan(token=13, start=113, end=114), TokenSpan(token=3, start=114, end=115)],
   [TokenSpan(token=10, start=116, end=117), TokenSpan(token=3, start=119, end=120)],
   [TokenSpan(token=1, start=124, end=125), TokenSpan(token=7, start=127, end=128)],
   [TokenSpan(token=7, start=129, end=130), TokenSpan(token=15, start=130, end=131), TokenSpan(token=2, start=132, end=133), TokenSpan(token=8, start=136, end=137)],
   [TokenSpan(token=10, start=141, end=142), TokenSpan(token=5, start=144, end=145), TokenSpan(token=10, start=148, end=149), TokenSpan(token=3, start=151, end=152), TokenSpan(token=4, start=153, end=154), TokenSpan(token=7, start=155, end=156)]
  ]


.. code-block:: python3

  def preview_word(waveform, spans, num_frames, transcript, sample_rate=bundle.sample_rate):
      ratio = waveform.size(1) / num_frames
      x0 = int(ratio * spans[0].start)
      x1 = int(ratio * spans[-1].end)
      print(f"{transcript} {x0 / sample_rate:.3f} - {x1 / sample_rate:.3f} sec")
      segment = waveform[:, x0:x1]
      return IPython.display.Audio(segment.numpy(), rate=sample_rate)
  num_frames = emission.size(1)

.. code-block:: python3

   preview_word(waveform, word_spans[0], num_frames, transcript[0])
   preview_word(waveform, word_spans[1], num_frames, transcript[1])
   preview_word(waveform, word_spans[2], num_frames, transcript[2])
   preview_word(waveform, word_spans[3], num_frames, transcript[3])
   preview_word(waveform, word_spans[4], num_frames, transcript[4])
   preview_word(waveform, word_spans[5], num_frames, transcript[5])
   preview_word(waveform, word_spans[6], num_frames, transcript[6])
   preview_word(waveform, word_spans[7], num_frames, transcript[7])
   preview_word(waveform, word_spans[8], num_frames, transcript[8])

The segmented wave of each word along with its time stamp is given below:

.. raw:: html

  <table>
    <tr>
      <th>Word</th>
      <th>Time</th>
      <th>Wave</th>
    </tr>
    <tr>
      <td>i</td>
      <td>0.644 - 0.664 sec</td>
      <td>
       <audio title="i.wav" controls="controls">
             <source src="/icefall/_static/kaldi-align/i.wav" type="audio/wav">
             Your browser does not support the <code>audio</code> element.
       </audio>
      </td>
    </tr>
    <tr>
      <td>had</td>
      <td>0.704 - 0.845 sec</td>
      <td>
       <audio title="had.wav" controls="controls">
             <source src="/icefall/_static/kaldi-align/had.wav" type="audio/wav">
             Your browser does not support the <code>audio</code> element.
       </audio>
      </td>
    </tr>
    <tr>
      <td>that</td>
      <td>0.885 - 1.026 sec</td>
      <td>
       <audio title="that.wav" controls="controls">
             <source src="/icefall/_static/kaldi-align/that.wav" type="audio/wav">
             Your browser does not support the <code>audio</code> element.
       </audio>
      </td>
    </tr>
    <tr>
      <td>curiosity</td>
      <td>1.086 - 1.790 sec</td>
      <td>
       <audio title="curiosity.wav" controls="controls">
             <source src="/icefall/_static/kaldi-align/curiosity.wav" type="audio/wav">
             Your browser does not support the <code>audio</code> element.
       </audio>
      </td>
    </tr>
    <tr>
      <td>beside</td>
      <td>1.871 - 2.314 sec</td>
      <td>
       <audio title="beside.wav" controls="controls">
             <source src="/icefall/_static/kaldi-align/beside.wav" type="audio/wav">
             Your browser does not support the <code>audio</code> element.
       </audio>
      </td>
    </tr>
    <tr>
      <td>me</td>
      <td>2.334 - 2.414 sec</td>
      <td>
       <audio title="me.wav" controls="controls">
             <source src="/icefall/_static/kaldi-align/me.wav" type="audio/wav">
             Your browser does not support the <code>audio</code> element.
       </audio>
      </td>
    </tr>
    <tr>
      <td>at</td>
      <td>2.495 - 2.575 sec</td>
      <td>
       <audio title="at.wav" controls="controls">
             <source src="/icefall/_static/kaldi-align/at.wav" type="audio/wav">
             Your browser does not support the <code>audio</code> element.
       </audio>
      </td>
    </tr>
    <tr>
      <td>this</td>
      <td>2.595 - 2.756 sec</td>
      <td>
       <audio title="this.wav" controls="controls">
             <source src="/icefall/_static/kaldi-align/this.wav" type="audio/wav">
             Your browser does not support the <code>audio</code> element.
       </audio>
      </td>
    </tr>
    <tr>
      <td>moment</td>
      <td>2.837 - 3.138 sec</td>
      <td>
       <audio title="moment.wav" controls="controls">
             <source src="/icefall/_static/kaldi-align/moment.wav" type="audio/wav">
             Your browser does not support the <code>audio</code> element.
       </audio>
      </td>
    </tr>
  </table>

We repost the whole wave below for ease of reference:

.. raw:: html

  <table>
    <tr>
      <th>Wave filename</th>
      <th>Content</th>
      <th>Text</th>
    </tr>
    <tr>
      <td>Lab41-SRI-VOiCES-src-sp0307-ch127535-sg0042.wav</td>
      <td>
       <audio title="Lab41-SRI-VOiCES-src-sp0307-ch127535-sg0042.wav" controls="controls">
             <source src="/icefall/_static/kaldi-align/Lab41-SRI-VOiCES-src-sp0307-ch127535-sg0042.wav" type="audio/wav">
             Your browser does not support the <code>audio</code> element.
       </audio>
      </td>
      <td>
        i had that curiosity beside me at this moment
      </td>
    </tr>
  </table>

Summary
-------

Congratulations! You have succeeded in using the FST-based approach to
compute alignment of a test wave.