# 一、马尔可夫性质

  马尔科夫性质——当前的状态只和上一时刻有关，在上一时刻之前的任何状态都和我无关。我们称其符合马尔可夫性质。

  具有马尔科夫性质的状态满足下面公式：

  $P(S_{t+1}|S_1,S_2,...,S_t) = P(S_{t+1}|S_t)$

  根据公式也就是说给定当前状态$S_t$，将来的状态与t时刻之前的状态已经没有关系。

# 二、马尔可夫链

  马尔可夫链是指具有马尔可夫性质的随机过程。在过程中，在给定当前信息的情况下，过去的信息状态对于预测将来状态是无关的。——可以称之为马尔科夫无后效性。

# 三、隐马尔可夫模型

  隐马尔可夫模型(Hidden Markov Model, HMM)是一种统计模型，在语音识别、行为识别、NLP、故障诊断等领域具有高效的性能。

  HMM是关于时序的概率模型，描述一个含有未知参数的马尔可夫链所生成的不可观测的状态随机序列，再由各个状态生成观测随机序列的过程。

  HMM是一个双重随机过程---具有一定状态的隐马尔可夫链和随机的观测序列。

  HMM随机生成的状态随机序列被称为状态序列；每个状态生成一个观测，由此产生的观测随机序列，被称为观测序列。

  举个例子来理解下hmm，我们使用输入法打字，敲出的每个字符就是观测序列，而实际我们想写的话就是隐藏序列。那不同公司的输入法都是在努力提高猜测咱们要写的话的能力。

## 1、定义

  对于HMM模型，首先我们假设Q是所有可能的隐藏状态的集合，V是所有可能的观测状态的集合，即：

$Q={q_1,q_2,...,q_N},V={v_1,v_2,...v_M}$

  其中，N是可能的隐藏状态数，M是所有的可能的观察状态数。

  对于一个长度为T的序列，I对应的状态序列, O是对应的观察序列，即：

$I={i_1,i_2,...,i_T},O={o_1,o_2,...o_T}$

  其中，任意一个隐藏状态$i_t∈Q$,任意一个观察状态$o_t∈V$

  HMM模型做了两个很重要的假设如下：

1. 齐次马尔科夫链假设。即任意时刻的隐藏状态只依赖于它前一个隐藏状态，这个我们在MCMC(二)马尔科夫链中有详细讲述。当然这样假设有点极端，因为很多时候我们的某一个隐藏状态不仅仅只依赖于前一个隐藏状态，可能是前两个或者是前三个。但是这样假设的好处就是模型简单，便于求解。如果在时刻t的隐藏状态是$i_t=q_i$,在时刻t+1的隐藏状态是$i_{t+1}=q_j$, 则从时刻t到时刻t+1的HMM状态转移概率$a_{ij}$可以表示为：

$a_{ij}=P(i_{t+1}=q_j|i_t=q_i)$

  这样$a_{ij}$可以组成马尔科夫链的状态转移矩阵A:

$A=[a_{ij}]_{N×N}$

2. 观测独立性假设。即任意时刻的观察状态只仅仅依赖于当前时刻的隐藏状态，这也是一个为了简化模型的假设。如果在时刻t的隐藏状态是$i_t=q_j$, 而对应的观察状态为$o_t=v_k$, 则该时刻观察状态$v_k$在隐藏状态$q_j$下生成的概率为$b_j(k)$,满足：

$b_j(k)=P(o_t=v_k|i_t=q_j)$

  这样$b_j(k)$可以组成观测状态生成的概率矩阵B:

$B=[b_j(k)]_{N×M}$

  除此之外，我们需要一组在时刻t=1的隐藏状态概率分布Π:

$Π=[π(i)]_N$  其中$π(i)=P(i_1=q_i)$

  一个HMM模型，可以由隐藏状态初始概率分布Π, 状态转移概率矩阵A和观测状态概率矩阵B决定。Π,A决定状态序列，B决定观测序列。因此，HMM模型可以由一个三元组λ表示如下：

$λ=(A,B,Π)$

## 2、hmm能解决的问题

1. 评估观察序列概率。即给定模型$λ=(A,B,Π)$和观测序列$O={o_1,o_2,...o_T}$，计算在模型$λ$下观测序列O出现的概率$P(O|λ)$。这个问题的求解需要用到前向后向算法。

2. 模型参数学习问题。即给定观测序列$O={o_1,o_2,...o_T}$，估计模型$λ=(A,B,Π)$的参数，使该模型下观测序列的条件概率$P(O|λ)$最大。这个问题的求解需要用到基于EM算法的鲍姆-韦尔奇算法。

3. 预测问题，也称为解码问题。即给定模型$λ=(A,B,Π)$和观测序列$O={o_1,o_2,...o_T}$，求给定观测序列条件下，最可能出现的对应的状态序列，这个问题的求解需要用到基于动态规划的维特比算法。【预测出的每一个隐状态都有N个值，每个值对应一个概率值；例如句子分词，那么就需要利用维特比算法找到整个句子的最大隐状态概率对应的隐状态路径。】

# 四、评估观察序列概率

## 1、求观测序列的概率

这个问题是这样的。我们已知HMM模型的参数𝜆=(𝐴,𝐵,Π)。其中𝐴是隐藏状态转移概率的矩阵，𝐵是观测状态生成概率的矩阵， Π是隐藏状态的初始概率分布。同时我们也已经得到了观测序列$O =\{o_1,o_2,...o_T\}$,现在我们要求观测序列𝑂在模型𝜆下出现的条件概率𝑃(𝑂|𝜆)。

乍一看，这个问题很简单。因为我们知道所有的隐藏状态之间的转移概率和所有从隐藏状态到观测状态生成概率，那么我们是可以暴力求解的。

我们可以列举出所有可能出现的长度为𝑇的隐藏序列$I = \{i_1,i_2,...,i_T\}$,分布求出这些隐藏序列与观测序列$O =\{o_1,o_2,...o_T\}$的联合概率分布𝑃(𝑂,𝐼|𝜆)，这样我们就可以很容易的求出边缘分布𝑃(𝑂|𝜆)了。

具体暴力求解的方法是这样的：首先，任意一个隐藏序列$I = \{i_1,i_2,...,i_T\}$出现的概率是：

$$P(I|\lambda) = \pi_{i_1} a_{i_{1}i_{2}} a_{i_{2}i_{3}}... a_{i_{T-1}\;\;i_T}$$

对于固定的状态序列$I = \{i_1,i_2,...,i_T\}$，我们要求的观察序列$O =\{o_1,o_2,...o_T\}$出现的概率是：

$$P(O|I, \lambda) = b_{i_1}(o_1)b_{i_2}(o_2)...b_{i_T}(o_T)$$

则𝑂和𝐼联合出现的概率是：

$$P(O,I|\lambda) = P(I|\lambda)P(O|I, \lambda) = \pi_{i_1}b_{i_1}(o_1)a_{i_1i_2}b_{i_2}(o_2)...a_{i_{T-1}\;\;i_T}b_{i_T}(o_T)$$

然后求边缘概率分布，即可得到观测序列𝑂在模型𝜆下出现的条件概率𝑃(𝑂|𝜆)，但是我们需要计算所有的T个隐状态、N种隐状态的自由组合，即$T^N$种组合的概率然后累加：

>边缘分布（Marginal Distribution）指在概率论和统计学的多维随机变量中，只包含其中部分变量的概率分布。\
>\
>**定义** \
>假设有一个和两个变量相关的概率分布：\
>$P(x,y)$\
>关于其中一个特定变量的边缘分布则为给定其他变量的条件概率分布：\
>$P(x)=\sum_yP(x,y)=\sum_yP(x|y)P(y)$ \
>解释：让y取所有的可能值，然后求出只有x变量的概率；举个例子：x为肤色（黑、白、黄）y为性别（男、女）那么$P(x)=P(x|男)+P(x|女)$ \
>参阅Wikipedia举例，下图中，X和Y遵从绿圈内所示的二元正态分布，红线和蓝线分别表示Y变量和X变量的边缘分布 \
>
>在这个边缘分布中，我们得到只关于一个变量的概率分布，而不再考虑另一变量的影响，实际上进行了降维操作。在实际应用中，例如人工神经网络的神经元互相关联，在计算它们各自的参数的时候，就会使用边缘分布计算得到某一特定神经元（变量）的值。

$$P(O|\lambda) = \sum\limits_{I}P(O,I|\lambda)  = \sum\limits_{i_1,i_2,...i_T}\pi_{i_1}b_{i_1}(o_1)a_{i_1i_2}b_{i_2}(o_2)...a_{i_{T-1}\;\;i_T}b_{i_T}(o_T)$$

虽然上述方法有效，但是如果我们的隐藏状态数𝑁非常多的那就麻烦了，此时我们预测状态有$N^T$种组合，算法的时间复杂度是$𝑂(𝑇𝑁^𝑇)$阶的。因此对于一些隐藏状态数极少的模型，我们可以用暴力求解法来得到观测序列出现的概率，但是如果隐藏状态多，则上述算法太耗时，我们需要寻找其他简洁的算法。

前向后向算法就是来帮助我们在较低的时间复杂度情况下求解这个问题

# 2、前向算法求HMM观测序列的概率

前向后向算法是前向算法和后向算法的统称，这两个算法都可以用来求HMM观测序列的概率。我们先来看看前向算法是如何求解这个问题的。

前向算法本质上属于动态规划的算法，也就是我们要通过找到局部状态递推的公式，这样一步步的从子问题的最优解拓展到整个问题的最优解。

在前向算法中，通过定义“前向概率”来定义动态规划的这个局部状态。什么是前向概率呢, 其实定义很简单：定义时刻𝑡时隐藏状态为$q_i$, 观测状态的序列为$𝑜_1,𝑜_2,...𝑜_𝑡$的概率为前向概率。记为：

$$\alpha_t(i) = P(o_1,o_2,...o_t, i_t =q_i | \lambda)\ \ \ \ 注意：字母是alpha不是a$$

既然是动态规划，我们就要递推了，现在我们假设我们已经找到了在时刻𝑡时各个隐藏状态的前向概率，现在我们需要递推出时刻𝑡+1时各个隐藏状态的前向概率。

从上图可以看出，我们可以基于时刻𝑡时各个隐藏状态的前向概率，再乘以对应的状态转移概率，即$\alpha_t(j)a_{ji}$就是在时刻𝑡观测到$o_1,o_2,...o_t$，并且时刻𝑡隐藏状态$𝑞_𝑗$, 时刻𝑡+1隐藏状态$𝑞_i$的概率。表示为：

$$P(o_1,o_2,...o_t,i_t=q_j,i_{t+1}=q_i|\lambda)=\alpha_t(j)a_{ji}$$

如果将想上面所有的线对应的概率求和，即$\sum\limits_{j=1}^N\alpha_t(j)a_{ji}$就是在时刻𝑡观测到$o_1,o_2,...o_t$，并且时刻𝑡+1隐藏状态$𝑞_i$的概率。表示为：

$$P(o_1,o_2,...o_t,i_{t+1}=q_i|\lambda)=\sum\limits_{j=1}^N\alpha_t(j)a_{ji}$$

继续一步，由于观测状态$O_{𝑡+1}$只依赖于𝑡+1时刻隐藏状态$𝑞_𝑖$, 这样$[\sum\limits_{j=1}^N\alpha_t(j)a_{ji}]b_i(o_{t+1})$就是在在时刻𝑡+1观测到$o_1,o_2,...o_t，o_{t+1}$，并且时刻𝑡+1隐藏状态𝑞𝑖的概率。表示为：

$$P(o_1,o_2,...o_t,o_{t+1},i_{t+1}=q_i|\lambda)=[\sum\limits_{j=1}^N\alpha_t(j)a_{ji}]b_i(o_{t+1})$$

而这个概率，恰恰就是时刻𝑡+1对应的隐藏状态𝑖的前向概率，这样我们得到了前向概率的递推关系式如下：

$$\alpha_{t+1}(i) = \Big[\sum\limits_{j=1}^N\alpha_t(j)a_{ji}\Big]b_i(o_{t+1})$$

我们的动态规划从时刻1开始，到时刻𝑇结束，由于$\alpha_T(i)$表示在时刻𝑇观测序列为$o_1,o_2,...o_T$，并且时刻𝑇隐藏状态$𝑞_𝑖$的概率，我们只要将所有隐藏状态对应的概率相加，即$\sum\limits_{i=1}^N\alpha_T(i)$就得到了在时刻𝑇观测序列为$o_1,o_2,...o_T$的概率。

下面总结下前向算法。

输入：HMM模型𝜆=(𝐴,𝐵,Π)，观测序列𝑂=($o_1,o_2,...o_T$)

输出：观测序列概率𝑃(𝑂|𝜆)

1) 计算时刻1的各个隐藏状态前向概率：

$$\alpha_1(i) = \pi_ib_i(o_1),\; i=1,2,...N$$

2) 递推时刻2,3,...𝑇时刻的前向概率（递推关系从时刻1往时刻T推，由t时刻的前向概率可以求出t+1时刻的前向概率）：

$$\alpha_{t+1}(i) = \Big[\sum\limits_{j=1}^N\alpha_t(j)a_{ji}\Big]b_i(o_{t+1}),\; i=1,2,...N$$

3) 计算最终结果：

$$P(O|\lambda) = \sum\limits_{i=1}^N\alpha_T(i)$$

从递推公式可以看出，我们的算法时间复杂度是$𝑂(𝑇𝑁^2)$，比暴力解法的时间复杂度$𝑂(𝑇𝑁^𝑇)$少了几个数量级。

## 3、HMM前向算法求解实例

我们的观察集合是:

$$𝑉={红，白}，𝑀=2$$

我们的状态集合是：

$$𝑄=\{盒子1，盒子2，盒子3\}，𝑁=3$$

而观察序列和状态序列的长度为3.

初始状态分布为：

$$\Pi = (0.2,0.4,0.4)^T$$

状态转移概率分布矩阵为：

$$A = \left( \begin{array} {ccc} 0.5 & 0.2 & 0.3 \\ 0.3 & 0.5 & 0.2 \\ 0.2 & 0.3 &0.5 \end{array} \right)$$

观测状态概率矩阵为：

$$B = \left( \begin{array} {ccc} 0.5 & 0.5 \\ 0.4 & 0.6 \\ 0.7 & 0.3 \end{array} \right)$$

球的颜色的观测序列:

$$O=\{红，白，红\}$$

首先计算时刻1三个状态的前向概率：

时刻1是红色球，隐藏状态是盒子1的概率为：

$$\alpha_1(1) = \pi_1b_1(o_1) = 0.2 \times 0.5 = 0.1$$

隐藏状态是盒子2的概率为：

$$\alpha_1(2) = \pi_2b_2(o_1) = 0.4 \times 0.4 = 0.16$$

隐藏状态是盒子3的概率为：

$$\alpha_1(3) = \pi_3b_3(o_1) = 0.4 \times 0.7 = 0.28$$

现在我们可以开始递推了，首先递推时刻2三个状态的前向概率：
时刻2是白色球，隐藏状态是盒子1的概率为：

$$\alpha_2(1) =  \Big[\sum\limits_{i=1}^3\alpha_1(i)a_{i1}\Big]b_1(o_2) = [0.1*0.5+0.16*0.3+0.28*0.2 ] \times 0.5 = 0.077$$

隐藏状态是盒子2的概率为：

$$\alpha_2(2) =  \Big[\sum\limits_{i=1}^3\alpha_1(i)a_{i2}\Big]b_2(o_2) = [0.1*0.2+0.16*0.5+0.28*0.3 ] \times 0.6 = 0.1104$$

隐藏状态是盒子3的概率为：

$$\alpha_2(3) =  \Big[\sum\limits_{i=1}^3\alpha_1(i)a_{i3}\Big]b_3(o_2) = [0.1*0.3+0.16*0.2+0.28*0.5 ] \times 0.3 = 0.0606$$

继续递推，现在我们递推时刻3三个状态的前向概率：
时刻3是红色球，隐藏状态是盒子1的概率为：

$$\alpha_3(1) =  \Big[\sum\limits_{i=1}^3\alpha_2(i)a_{i1}\Big]b_1(o_3) = [0.077*0.5+0.1104*0.3+0.0606*0.2 ] \times 0.5 = 0.04187$$

隐藏状态是盒子2的概率为：

$$\alpha_3(2) =  \Big[\sum\limits_{i=1}^3\alpha_2(i)a_{i2}\Big]b_2(o_3) = [0.077*0.2+0.1104*0.5+0.0606*0.3 ] \times 0.4 = 0.03551$$

隐藏状态是盒子3的概率为：

$$\alpha_3(3) =  \Big[\sum\limits_{i=1}^3\alpha_2(i)a_{i3}\Big]b_3(o_3) = [0.077*0.3+0.1104*0.2+0.0606*0.5 ] \times 0.7 = 0.05284$$

最终我们求出观测序列:$𝑂=\{红，白，红\}$的概率为：

$$P(O|\lambda) = \sum\limits_{i=1}^3\alpha_3(i) = 0.13022$$

## 4、后向算法求HMM观测序列的概率

后向算法和前向算法非常类似，都是用的动态规划，唯一的区别是选择的局部状态不同，后向算法用的是“后向概率”，那么后向概率是如何定义的呢？

定义时刻𝑡时隐藏状态为$𝑞_𝑖$, 从时刻𝑡+1到最后时刻𝑇的观测状态的序列为$𝑜_{𝑡+1},𝑜_{𝑡+2},...𝑜_𝑇$的概率为后向概率。记为：

$$\beta_t(i) = P(o_{t+1},o_{t+2},...o_T| i_t =q_i , \lambda)$$

后向概率的动态规划递推公式和前向概率是相反的。现在我们假设我们已经找到了在时刻𝑡+1时各个隐藏状态的后向概率$𝛽_{𝑡+1}(𝑗)$，表示为：

$$\beta_{t+1}(j) = P(o_{t+2},o_{t+3}...o_T| i_{t+1} =q_j , \lambda)$$

现在我们需要递推出时刻𝑡时各个隐藏状态的后向概率。如上图，我们可以计算出观测状态的序列为$o_{𝑡+2},o_{𝑡+3},...o_𝑇$， 𝑡时隐藏状态为$𝑞_𝑖$, 时刻𝑡+1隐藏状态为$𝑞_𝑗$的概率为$𝑎_{𝑖𝑗}𝛽_{𝑡+1}(𝑗)$, 表示为：

$$P(o_{𝑡+2},o_{𝑡+3},...o_𝑇|i_t=q_i,i_{t+1}=q_j,\lambda)=𝑎_{𝑖𝑗}𝛽_{𝑡+1}(𝑗)$$

接着可以得到观测状态的序列为$o_{𝑡+1},o_{𝑡+2},...o_𝑇$， 𝑡时隐藏状态为$𝑞_𝑖$, 时刻𝑡+1隐藏状态为$𝑞_𝑗$的概率为$a_{ij}b_j(o_{t+1})\beta_{t+1}(j)$, 表示为：

$$P(o_{𝑡+1},o_{𝑡+2},...o_𝑇|i_t=q_i,i_{t+1}=q_j,\lambda)=a_{ij}b_j(o_{t+1})\beta_{t+1}(j)$$

则把上面所有线对应的概率加起来，我们可以得到观测状态的序列为$o_{𝑡+1},o_{𝑡+2},...o_𝑇$， 𝑡时隐藏状态为$𝑞_𝑖$的概率为$\sum\limits_{j=1}^{N}a_{ij}b_j(o_{t+1})\beta_{t+1}(j)$，这个概率即为时刻𝑡的后向概率。表示为：

$$P(o_{𝑡+1},o_{𝑡+2},...o_𝑇|i_t=q_i,\lambda)=\sum\limits_{j=1}^{N}a_{ij}b_j(o_{t+1})\beta_{t+1}(j)$$

这样我们得到了后向概率的递推关系式如下：
$$\beta_{t}(i) = \sum\limits_{j=1}^{N}a_{ij}b_j(o_{t+1})\beta_{t+1}(j)$$
现在我们总结下后向算法的流程,注意下和前向算法的相同点和不同点：

输入：HMM模型𝜆=(𝐴,𝐵,Π)，观测序列$O=(o_1,o_2,...o_T)$

输出：观测序列概率𝑃(𝑂|𝜆)

1) 初始化时刻𝑇的各个隐藏状态后向概率：

$$\beta_T(i) = 1,\; i=1,2,...N$$

2) 递推时刻𝑇−1,𝑇−2,...1时刻的后向概率（递推关系从T时刻往T-1,T-2...1时刻推，利用t+1时刻的后向概率可以求t时刻的后向概率）：

$$\beta_{t}(i) = \sum\limits_{j=1}^{N}a_{ij}b_j(o_{t+1})\beta_{t+1}(j),\; i=1,2,...N$$

3) 计算最终结果：

$$P(O|\lambda) = \sum\limits_{i=1}^N\pi_ib_i(o_1)\beta_1(i)$$

此时我们的算法时间复杂度仍然是$𝑂(𝑇𝑁^2)$。

## 5、HMM常用概率的计算

利用前向概率和后向概率，我们可以计算出HMM中单个状态和两个状态的概率公式。

1.给定模型𝜆和观测序列𝑂,在时刻𝑡处于状态$𝑞_𝑖$的概率记为:

$$\gamma_t(i) = P(i_t = q_i | O,\lambda) = \frac{P(i_t = q_i ,O|\lambda)}{P(O|\lambda)}$$

那么$P(i_t = q_i ,O|\lambda)$怎么求呢？

我们先分析下这个概率求的是啥，

它的意思是:在给定HMM的参数λ，观测序列是$O(O_1,O_2,...O_T)$,在t时刻的状态为$q_i$的概率。

换句话理解就是:从1到t时刻，t时刻的状态为$q_i$,输出是$O(O_1,O_2,...O_T)$的概率P1；然后t时刻之后到T，在t时刻状态为$q_i$的基础上，输出序列是$o_{t+1},o_{t+2}..o_T$的概率P2，P1和P2相乘即为所求。

碰巧P1符合前向概率定义，P2符合后向概率定义，所以 $P1=\alpha_t(i) \ \ \  P2=\beta_t(i)$，则:

$$P(i_t = q_i ,O|\lambda) = P1*P2 = \alpha_t(i)\beta_t(i)$$

那么：

$$P(O|\lambda) = \sum_j^N \alpha_t(j)\beta_t(j) \ \ \ 注意：为啥P(O|\lambda)用这个表示，因为这样方便计算$$

所以：

$$\gamma_t(i) = \frac{ \alpha_t(i)\beta_t(i)}{\sum\limits_{j=1}^N \alpha_t(j)\beta_t(j)}$$

2.给定模型𝜆和观测序列𝑂,在时刻𝑡处于状态$𝑞_𝑖$，且时刻𝑡+1处于状态$𝑞_𝑗$的概率记为:

$$\xi_t(i,j) = P(i_t = q_i, i_{t+1}=q_j | O,\lambda) = \frac{ P(i_t = q_i, i_{t+1}=q_j , O|\lambda)}{P(O|\lambda)}$$

那么$P(i_t = q_i, i_{t+1}=q_j , O|\lambda)$怎么求呢？

我们先分析下这个概率求的是啥，

它的意思是：在给定HMM的参数下，观测序列是O，t时刻的状态是$q_i$，t+1时刻的状态是$q_j$的概率。

这个概率=$P(O_1,O_2,...O_t,i_t=q_i|λ)*P(q_i转移到t+1时刻的q_j)*P(o_{t+2},o_{t+3}..o_T|i_{t+1}=q_j, λ)*P(t+1时刻输出是O_{t+1})$

逐个解释下：

>$P(O_1,O_2,...O_t,i_t=q_i|λ)$:这不就是t时刻的前向概率$\alpha_t(i)$么\
>$P(q_i转移到t+1时刻的q_j)$:这不就是转移概率$a_{ij}$么\
>$P(o_{t+2},o_{t+3}..o_T|i_{t+1}=q_j, λ)$:这不就是t+1时刻的后向概率$\beta_{t+1}(i)$么\
>(注意:t+1时刻的后续输出序列是$o_{t+2},o_{t+3}..o_T$,即下标是从t+2开始，不是t+1开始，请仔细理解后向概率的定义)。\
>前面少了$O_{t+1}$这个时间点的观测序列，因此要补上$P(t+1时刻输出是O_{t+1})$,而这个不就是$b_{j}(O_{t+1})$么

所以：

$$P(i_t = q_i, i_{t+1}=q_j , O|\lambda) = \alpha_t(i)a_{ij}b_j(o_{t+1})\beta_{t+1}(j)$$

所以：

$$P(O|\lambda)=\sum\limits_{r=1}^N\sum\limits_{s=1}^N\alpha_t(r)a_{rs}b_s(o_{t+1})\beta_{t+1}(s)$$

所以：

$$\xi_t(i,j) = \frac{\alpha_t(i)a_{ij}b_j(o_{t+1})\beta_{t+1}(j)}{\sum\limits_{r=1}^N\sum\limits_{s=1}^N\alpha_t(r)a_{rs}b_s(o_{t+1})\beta_{t+1}(s)}$$

3.将$𝛾_𝑡(𝑖)$和$𝜉_𝑡(𝑖,𝑗)$在各个时刻𝑡求和，可以得到：

在观测序列𝑂下状态𝑖出现的期望值$\sum_{𝑡=1}^𝑇𝛾_𝑡(𝑖)$

在观测序列𝑂下由状态𝑖转移的期望值$\sum_{𝑡=1}^{𝑇-1}𝛾_𝑡(𝑖)$

在观测序列𝑂下由状态𝑖转移到状态𝑗的期望值$\sum_{𝑡=1}^{𝑇-1}𝜉_𝑡(𝑖,𝑗)$

# 五、模型参数学习问题

## 1、HMM模型参数求解概述

HMM模型参数求解根据已知的条件可以分为两种情况。

第一种情况较为简单，就是我们已知𝐷个长度为𝑇的观测序列和对应的隐藏状态序列，即${(𝑂_1,𝐼_1),(𝑂_2,𝐼_2),...(𝑂_𝐷,𝐼_𝐷)}$是已知的，此时我们可以很容易的用最大似然来求解模型参数。

假设样本从隐藏状态$𝑞_𝑖$转移到$𝑞_𝑗$的频率计数是$𝐴_{𝑖𝑗}$,那么状态转移矩阵求得为：

$$A = \Big[a_{ij}\Big], \;其中a_{ij} = \frac{A_{ij}}{\sum\limits_{s=1}^{N}A_{is}}$$

假设样本隐藏状态为$𝑞_𝑗$且观测状态为$𝑣_𝑘$的频率计数是$𝐵_{𝑗𝑘}$,那么观测状态概率矩阵为：

$$B= \Big[b_{j}(k)\Big], \;其中b_{j}(k) = \frac{B_{jk}}{\sum\limits_{s=1}^{M}B_{js}}$$

假设所有样本中初始隐藏状态为$𝑞_𝑖$的频率计数为$𝐶(𝑖)$,那么初始概率分布为：

$$\Pi = \pi(i) = \frac{C(i)}{\sum\limits_{s=1}^{N}C(s)}$$

可见第一种情况下求解模型还是很简单的。但是在很多时候，我们无法得到HMM样本观察序列对应的隐藏序列，只有𝐷个长度为𝑇的观测序列，即${(𝑂_1),(𝑂_2),...(𝑂_𝐷)}$是已知的，此时我们能不能求出合适的HMM模型参数呢？这就是我们的第二种情况，也是我们本文要讨论的重点。它的解法最常用的是鲍姆-韦尔奇算法，其实就是基于EM算法的求解，只不过鲍姆-韦尔奇算法出现的时代，EM算法还没有被抽象出来，所以我们本文还是说鲍姆-韦尔奇算法。

## 2、鲍姆-韦尔奇（Baum-Welch）算法原理

　鲍姆-韦尔奇算法原理既然使用的就是EM算法【（Expectation-Maximum）算法也称期望最大化算法】的原理，那么我们需要在E步求出联合分布𝑃(𝑂,𝐼|𝜆)基于条件概率$P(I|O,\overline{\lambda})$的期望，其中$\overline{\lambda}$为当前的模型参数，然后再M步最大化这个期望，得到更新的模型参数𝜆。接着不停的进行EM迭代，直到模型参数的值收敛为止。

首先来看看E步，当前模型参数为$\overline{\lambda}$, 联合分布𝑃(𝑂,𝐼|𝜆)基于条件概率$P(I|O,\overline{\lambda})$的期望表达式为：

$$L(\lambda, \overline{\lambda}) = \sum\limits_{I}P(I|O,\overline{\lambda})logP(O,I|\lambda)$$

在M步，我们极大化上式，然后得到更新后的模型参数如下：

$$\overline{\lambda} = arg\;\max_{\lambda}\sum\limits_{I}P(I|O,\overline{\lambda})logP(O,I|\lambda)$$

通过不断的E步和M步的迭代，直到$\overline{\lambda}$收敛。下面我们来看看鲍姆-韦尔奇算法的推导过程。

## 3、鲍姆-韦尔奇算法的推导

我们的训练数据为$\{(O_1, I_1), (O_2, I_2), ...(O_D, I_D)\}$，其中任意一个观测序列$O_d = \{o_1^{(d)}, o_2^{(d)}, ... o_T^{(d)}\}$,其对应的未知的隐藏状态序列表示为：$I_d = \{i_1^{(d)}, i_2^{(d)}, ... i_T^{(d)}\}$

首先看鲍姆-韦尔奇算法的E步，我们需要先计算联合分布$𝑃(𝑂,𝐼|\lambda)$的表达式如下：

$$P(O,I|\lambda) = \prod_{d=1}^D \pi_{i_1^{(d)}}\ b_{i_1^{(d)}}\ (o_1^{(d)})\ a_{i_1^{(d)}\ i_2^{(d)}}\ b_{i_2^{(d)}}\ (o_2^{(d)}) ... a_{i_{T-1}^{(d)} \ \ i_T^{(d)}}\ b_{i_T^{(d)}}\ \ (o_T^{(d)})$$

我们的E步得到的期望表达式为：

$$L(\lambda, \overline{\lambda}) = \sum\limits_{I}P(I|O,\overline{\lambda})logP(O,I|\lambda)$$

在M步我们要极大化上式。由于$P(I|O,\overline{\lambda}) = \frac{P(I,O|\overline{\lambda})}{P(O|\overline{\lambda})}$,而$𝑃(𝑂|\overline{\lambda})$是常数，因此我们要极大化的式子等价于：

$$\overline{\lambda} = arg\;\max_{\lambda}\sum\limits_{I}P(O,I|\overline{\lambda})logP(O,I|\lambda)$$

我们将上面$𝑃(𝑂,𝐼|𝜆)$的表达式带入我们的极大化式子，得到的表达式如下：

$$\overline{\lambda} = arg\;\max_{\lambda}\sum\limits_{d=1}^D\sum\limits_{I}P(O,I|\overline{\lambda})(log\pi_{i_1} + \sum\limits_{t=1}^{T-1}log\;a_{i_t,i_{t+1}} +  \sum\limits_{t=1}^Tlog b_{i_t}(o_t))$$

我们的隐藏模型参数$𝜆=(𝐴,𝐵,Π)$,因此下面我们只需要对上式分别对$𝐴,𝐵,Π$求导即可得到我们更新的模型参数$\overline{\lambda}$

首先我们看看对模型参数Π的求导。由于Π只在上式中括号里的第一部分出现，因此我们对于Π的极大化式子为：

$$\overline{\pi_i} = arg\;\max_{\pi_{i_1}} \sum\limits_{d=1}^D\sum\limits_{I}P(O,I|\overline{\lambda})log\pi_{i_1} = arg\;\max_{\pi_{i}} \sum\limits_{d=1}^D\sum\limits_{i=1}^NP(O,i_1^{(d)} =i|\overline{\lambda})log\pi_{i}$$

由于$𝜋_𝑖$还满足$\sum\limits_{i=1}^N\pi_i =1$，因此根据拉格朗日子乘法，我们得到𝜋𝑖要极大化的拉格朗日函数为：

$$arg\;\max_{\pi_{i}}\sum\limits_{d=1}^D\sum\limits_{i=1}^NP(O,i_1^{(d)} =i|\overline{\lambda})log\pi_{i} + \gamma(\sum\limits_{i=1}^N\pi_i -1)$$

其中，𝛾为拉格朗日系数。上式对$𝜋_𝑖$求偏导数并令结果为0， 我们得到：

$$\sum\limits_{d=1}^DP(O,i_1^{(d)} =i|\overline{\lambda}) + \gamma\pi_i = 0$$

令𝑖分别等于从1到𝑁，从上式可以得到𝑁个式子，对这𝑁个式子求和可得：

$$\sum\limits_{d=1}^DP(O|\overline{\lambda}) + \gamma = 0$$

从上两式消去𝛾,得到$𝜋_𝑖$的表达式为：

$$\pi_i =\frac{\sum\limits_{d=1}^DP(O,i_1^{(d)} =i|\overline{\lambda})}{\sum\limits_{d=1}^DP(O|\overline{\lambda})} = \frac{\sum\limits_{d=1}^DP(O,i_1^{(d)} =i|\overline{\lambda})}{DP(O|\overline{\lambda})} = \frac{\sum\limits_{d=1}^DP(i_1^{(d)} =i|O, \overline{\lambda})}{D} =  \frac{\sum\limits_{d=1}^DP(i_1^{(d)} =i|O^{(d)}, \overline{\lambda})}{D}$$

利用前向概率的定义可得：

$$P(i_1^{(d)} =i|O^{(d)}, \overline{\lambda}) = \gamma_1^{(d)}(i)$$

因此最终我们在M步$𝜋_𝑖$的迭代公式为：

$$\pi_i =  \frac{\sum\limits_{d=1}^D\gamma_1^{(d)}(i)}{D}$$

现在我们来看看𝐴的迭代公式求法。方法和Π的类似。由于𝐴只在最大化函数式中括号里的第二部分出现，而这部分式子可以整理为：

$$\sum\limits_{d=1}^D\sum\limits_{I}\sum\limits_{t=1}^{T-1}P(O,I|\overline{\lambda})log\;a_{i_t,i_{t+1}} = \sum\limits_{d=1}^D\sum\limits_{i=1}^N\sum\limits_{j=1}^N\sum\limits_{t=1}^{T-1}P(O,i_t^{(d)} = i, i_{t+1}^{(d)} = j|\overline{\lambda})log\;a_{ij}$$

由于$𝑎_{𝑖𝑗}$还满足$\sum\limits_{j=1}^Na_{ij} =1$。和求解𝜋𝑖类似，我们可以用拉格朗日子乘法并对$𝑎_{𝑖𝑗}$求导，并令结果为0，可以得到$𝑎_{𝑖𝑗}$的迭代表达式为：

$$a_{ij} = \frac{\sum\limits_{d=1}^D\sum\limits_{t=1}^{T-1}P(O^{(d)}, i_t^{(d)} = i, i_{t+1}^{(d)} = j|\overline{\lambda})}{\sum\limits_{d=1}^D\sum\limits_{t=1}^{T-1}P(O^{(d)}, i_t^{(d)} = i|\overline{\lambda})}$$

利用前向概率的定义和$𝜉𝑡(𝑖,𝑗)$的定义可得到在M步$𝑎_{𝑖𝑗}$的迭代公式为：

$$a_{ij} = \frac{\sum\limits_{d=1}^D\sum\limits_{t=1}^{T-1}\xi_t^{(d)}(i,j)}{\sum\limits_{d=1}^D\sum\limits_{t=1}^{T-1}\gamma_t^{(d)}(i)}$$

现在我们来看看𝐵的迭代公式求法。方法和Π的类似。由于𝐵只在最大化函数式中括号里的第三部分出现，而这部分式子可以整理为：

$$\sum\limits_{d=1}^D\sum\limits_{I}\sum\limits_{t=1}^{T}P(O,I|\overline{\lambda})log\;b_{i_t}(o_t) = \sum\limits_{d=1}^D\sum\limits_{j=1}^N\sum\limits_{t=1}^{T}P(O,i_t^{(d)} = j|\overline{\lambda})log\;b_{j}(o_t)$$

由于$𝑏_𝑗(𝑜_𝑡)$还满足$\sum_{𝑘=1}^𝑀𝑏_𝑗(𝑜_𝑡=𝑣_𝑘)=1$。和求解$𝜋_𝑖$类似，我们可以用拉格朗日子乘法并对$𝑏_𝑗(𝑘)$求导，并令结果为0，得到$𝑏_𝑗(𝑘)$的迭代表达式为：

$$b_{j}(k) = \frac{\sum\limits_{d=1}^D\sum\limits_{t=1}^{T}P(O,i_t^{(d)} = j|\overline{\lambda})I(o_t^{(d)}=v_k)}{\sum\limits_{d=1}^D\sum\limits_{t=1}^{T}P(O,i_t^{(d)} = j|\overline{\lambda})}$$

其中$𝐼(𝑜_t^{(𝑑)}=𝑣_𝑘)$当且仅当$𝑜_t^{(𝑑)}=𝑣_𝑘$时为1，否则为0. 利用前向概率的定义可得$𝑏_𝑗(𝑜_𝑡)$的最终表达式为：

$$b_{j}(k) = \frac{\sum\limits_{d=1}^D\sum\limits_{t=1, o_t^{(d)}=v_k}^{T}\gamma_t^{(d)}(j)}{\sum\limits_{d=1}^D\sum\limits_{t=1}^{T}\gamma_t^{(d)}(j)}$$

有了$𝜋_𝑖,𝑎_{𝑖𝑗},𝑏_𝑗(𝑘)$的迭代公式，我们就可以迭代求解HMM模型参数了。

## 4、鲍姆-韦尔奇算法流程总结

概括总结下鲍姆-韦尔奇算法的流程。

**输入：** 𝐷个观测序列样本${(𝑂_1),(𝑂_2),...(𝑂_𝐷)}$

**输出：** HMM模型参数

1)随机初始化所有的$𝜋_𝑖,𝑎_{𝑖𝑗},𝑏_𝑗(𝑘)$

2)对于每个样本𝑑=1,2,...𝐷，用前向后向算法计算$\gamma_t^{(d)}(i)，\xi_t^{(d)}(i,j), t =1,2...T$

3)更新模型参数：

$$\pi_i =  \frac{\sum\limits_{d=1}^D\gamma_1^{(d)}(i)}{D}$$

$$a_{ij} = \frac{\sum\limits_{d=1}^D\sum\limits_{t=1}^{T-1}\xi_t^{(d)}(i,j)}{\sum\limits_{d=1}^D\sum\limits_{t=1}^{T-1}\gamma_t^{(d)}(i)}$$

$$b_{j}(k) = \frac{\sum\limits_{d=1}^D\sum\limits_{t=1, o_t^{(d)}=v_k}^{T}\gamma_t^{(d)}(j)}{\sum\limits_{d=1}^D\sum\limits_{t=1}^{T}\gamma_t^{(d)}(j)}$$

4)如果$𝜋_𝑖,𝑎_{𝑖𝑗,}𝑏_𝑗(𝑘)$的值已经收敛，则算法结束，否则回到第2）步继续迭代。

# 六、预测问题

## 1、HMM最可能隐藏状态序列求解概述

在HMM模型的解码问题中，给定模型$𝜆=(𝐴,𝐵,Π)$和观测序列$𝑂={𝑜_1,𝑜_2,...𝑜_𝑇}$，求给定观测序列O条件下，最可能出现的对应的状态序列$I^*= \{i_1^*,i_2^*,...i_T^*\}$,即$𝑃(𝐼^∗|𝑂)$要最大化。

一个可能的近似解法是求出观测序列𝑂在每个时刻𝑡最可能的隐藏状态$𝑖_𝑡^*$然后得到一个近似的隐藏状态序列$I^*= \{i_1^*,i_2^*,...i_T^*\}$。要这样近似求解不难，利用前向后向算法评估观察序列概率的定义：在给定模型𝜆和观测序列𝑂时，在时刻𝑡处于状态$𝑞_𝑖$的概率是$𝛾_𝑡(𝑖)$，这个概率可以通过HMM的前向算法与后向算法计算。这样我们有：

$$i_t^* = arg \max_{1 \leq i \leq N}[\gamma_t(i)], \; t =1,2,...T$$

近似算法很简单，但是却不能保证预测的状态序列是整体是最可能的状态序列，因为预测的状态序列中某些相邻的隐藏状态可能存在转移概率为0的情况【也就是这样求解没有考虑到转移概率】。

而维特比算法可以将HMM的状态序列作为一个整体来考虑，避免近似算法的问题【考虑转移概率】，下面我们来看看维特比算法进行HMM解码的方法。

## 2、维特比算法概述

维特比算法是一个通用的解码算法，是基于动态规划的求序列最短路径的方法。

既然是动态规划算法，那么就需要找到合适的局部状态，以及局部状态的递推公式。在HMM中，维特比算法定义了两个局部状态用于递推。

第一个局部状态是在时刻𝑡隐藏状态为𝑖所有可能的状态转移路径$i_1,i_2,...i_t$中的概率最大值。记为$𝛿_𝑡(𝑖)$:

$$\delta_t(i) = \max_{i_1,i_2,...i_{t-1}}\;P(i_t=i, i_1,i_2,...i_{t-1},o_t,o_{t-1},...o_1|\lambda),\; i =1,2,...N$$

由$𝛿_𝑡(𝑖)$的定义可以得到𝛿的递推表达式：

$$
\begin{align}
\delta_{t+1}(i) & =  \max_{i_1,i_2,...i_{t}}\;P(i_{t+1}=i, i_1,i_2,...i_{t},o_{t+1},o_{t},...o_1|\lambda) \tag{1}\\
& = \max_{1 \leq j \leq N}\;[\delta_t(j)a_{ji}]b_i(o_{t+1}) \tag{2}
\end{align}
$$

第二个局部状态由第一个局部状态递推得到。我们定义在时刻𝑡隐藏状态为𝑖的所有单个状态转移路径$(i_1,i_2,...,i_{t-1},i)$中概率最大的转移路径中第𝑡−1个节点的隐藏状态为$Ψ_𝑡(𝑖)$,其递推表达式可以表示为：

$$\Psi_t(i) = arg \; \max_{1 \leq j \leq N}\;[\delta_{t-1}(j)a_{ji}]$$

有了这两个局部状态，我们就可以从时刻0一直递推到时刻𝑇，然后利用Ψ𝑡(𝑖)记录的前一个最可能的状态节点回溯，直到找到最优的隐藏状态序列。

## 3、维特比算法流程总结

总结下维特比算法的流程：

**输入：** HMM模型𝜆=(𝐴,𝐵,Π)，观测序列$O=(o_1,o_2,...o_T)$

**输出：** 最有可能的隐藏状态序列$I^*= \{i_1^*,i_2^*,...i_T^*\}$

1）初始化局部状态【时刻1的各个隐藏状态前向概率和上一个时刻的所有最可能的隐状态】：

$$\delta_1(i) = \pi_ib_i(o_1),\;i=1,2...N$$

$$\Psi_1(i)=0,\;i=1,2...N$$

2)进行动态规划递推时刻𝑡=2,3,...𝑇时刻的局部状态：

$$\delta_{t}(i) = \max_{1 \leq j \leq N}\;[\delta_{t-1}(j)a_{ji}]b_i(0_{t}),\;i=1,2...N$$

$$\Psi_t(i) = arg \; \max_{1 \leq j \leq N}\;[\delta_{t-1}(j)a_{ji}],\;i=1,2...N$$

3)计算时刻𝑇最大的$𝛿_𝑇(𝑖)$,即为最可能隐藏状态序列出现的概率。计算时刻𝑇最大的$Ψ_𝑡(𝑖)$,即为时刻𝑇最可能的隐藏状态。

$$P* = \max_{1 \leq j \leq N}\delta_{T}(i)$$

$$i_T^* = arg \; \max_{1 \leq j \leq N}\;[\delta_{T}(i)]$$

4)利用局部状态Ψ(𝑖)开始回溯。对于𝑡=𝑇−1,𝑇−2,...,1：

$$i_t^* = \Psi_{t+1}(i_{t+1}^*)$$

最终得到最有可能的隐藏状态序列$I^*= \{i_1^*,i_2^*,...i_T^*\}$

## 4、HMM维特比算法求解实例

我们的观察集合是:

$$𝑉={红，白}，𝑀=2$$

我们的状态集合是：

$$𝑄=\{盒子1，盒子2，盒子3\}，𝑁=3$$

而观察序列和状态序列的长度为3.

初始状态分布为：

$$\Pi = (0.2,0.4,0.4)^T$$

状态转移概率分布矩阵为：

$$A = \left( \begin{array} {ccc} 0.5 & 0.2 & 0.3 \\ 0.3 & 0.5 & 0.2 \\ 0.2 & 0.3 &0.5 \end{array} \right)$$

观测状态概率矩阵为：

$$B = \left( \begin{array} {ccc} 0.5 & 0.5 \\ 0.4 & 0.6 \\ 0.7 & 0.3 \end{array} \right)$$

球的颜色的观测序列:

$$O=\{红，白，红\}$$

首先计算时刻1三个状态的前向概率：

首先需要得到三个隐藏状态在时刻1时对应的各自两个局部状态，此时观测状态为1：

$$𝛿_1(1)=𝜋_1𝑏_1(𝑜_1)=0.2×0.5=0.1$$
$$𝛿_1(2)=𝜋_2𝑏_2(𝑜_1)=0.4×0.4=0.16$$
$$𝛿_1(3)=𝜋_3𝑏_3(𝑜_1)=0.4×0.7=0.28$$
$$Ψ_1(1)=Ψ_1(2)=Ψ_1(3)=0$$

现在开始递推三个隐藏状态在时刻2时对应的各自两个局部状态，此时观测状态为2：

$$\delta_2(1) = \max_{1\leq j \leq 3}[\delta_1(j)a_{j1}]b_1(o_2) = \max_{1\leq j \leq 3}[0.1 \times 0.5, 0.16 \times 0.3, 0.28\times 0.2] \times 0.5 = 0.028$$

$$\Psi_2(1)=3$$

$$\delta_2(2) = \max_{1\leq j \leq 3}[\delta_1(j)a_{j2}]b_2(o_2) = \max_{1\leq j \leq 3}[0.1 \times 0.2, 0.16 \times 0.5, 0.28\times 0.3] \times 0.6 = 0.0504$$

$$\Psi_2(2)=2$$

$$\delta_2(3) = \max_{1\leq j \leq 3}[\delta_1(j)a_{j3}]b_3(o_2) = \max_{1\leq j \leq 3}[0.1 \times 0.3, 0.16 \times 0.2, 0.28\times 0.5] \times 0.3 = 0.042$$

$$\Psi_2(3)=3$$

继续递推三个隐藏状态在时刻3时对应的各自两个局部状态，此时观测状态为1：

$$\delta_3(1) = \max_{1\leq j \leq 3}[\delta_2(j)a_{j1}]b_1(o_3) = \max_{1\leq j \leq 3}[0.028 \times 0.5, 0.0504 \times 0.3, 0.042\times 0.2] \times 0.5 = 0.00756$$

$$\Psi_3(1)=2$$

$$\delta_3(2) = \max_{1\leq j \leq 3}[\delta_2(j)a_{j2}]b_2(o_3) = \max_{1\leq j \leq 3}[0.028  \times 0.2, 0.0504\times 0.5, 0.042\times 0.3] \times 0.4 = 0.01008$$

$$\Psi_3(2)=2$$

$$\delta_3(3) = \max_{1\leq j \leq 3}[\delta_2(j)a_{j3}]b_3(o_3) = \max_{1\leq j \leq 3}[0.028  \times 0.3, 0.0504 \times 0.2, 0.042\times 0.5] \times 0.7 = 0.0147$$

$$\Psi_3(3)=3$$

此时已经到最后的时刻，我们开始准备回溯。此时最大概率为$𝛿_3(3)$,从而得到$𝑖_3^*=3$

由于$Ψ_3(3)=3$,所以$𝑖_2^*=3$, 而又由于$Ψ_2(3)=3$,所以$𝑖_1^*=3$。从而得到最终的最可能的隐藏状态序列为：$(3,3,3)$

## 5、HMM模型维特比算法总结

维特比算法也是寻找序列最短路径的一个通用方法，和dijkstra算法有些类似，但是dijkstra算法并没有使用动态规划，而是贪心算法。同时维特比算法仅仅局限于求序列最短路径，而dijkstra算法是通用的求最短路径的方法。

```python

```


```python
import numpy as np
%load_ext autoreload
```

Notebook by **Maxime Dion** <maxime.dion@usherbrooke.ca>

For the QSciTech-QuantumBC virtual workshop on gate-based quantum computing

## Tutorial for  Activity 3.2

For this activity, make sure you can easily import your versions of `hamiltonian.py`, `pauli_string.py` and `mapping.py` that you have completed in the Activity 3.1 tutorial. You will also need your verions of `evaluator.py` and `solver.py`. Placing this notebook in the same `path` as these files is the easiest way to acheive this. At the end of this notebook, you should be in good position to complete these 2 additionnal files.

The solution we suggest here is NOT mandatory. If you find ways to make it better and more efficient, go on and impress us! On the other hand, by completing all sections of this notebook you'll be able to :
- Prepare a Quantum State based on a varitional form (circuit);
- Measure qubits in the X, Y and Z basis;
- Estimate expectation value of Pauli String on a quantum state;
- Evaluate the expectation value of an Hamiltonian in the form of a Linear Combinaison of Pauli Strings;
- Run a minimization algorithm on the energy expectation function to find the ground state of a Hamiltonian;
- Dance to express your overwhelming sense of accomplishment

**Important**

When you modify and save a `*.py` file you need to re-import it so that your modifications can be taken into account when you re-execute a call. By adding the magic command `%autoreload` at the beginning of a cell, you make sure that the modifications you did to the `*.py` files are taken into account when you re-run a celll and that you can see the effect.

If you encounter unusual results, restart the kernel and try again.

**Note on numbering**

When you ask a question in the Slack channel you can refer to the section name or the section number.

To enable the section numbering, please make sure you install [nbextensions](https://jupyter-contrib-nbextensions.readthedocs.io/en/latest/install.html). It is available in the conda distribution. After installation it you need to enable the option 'Table of contents (2)'.

# Variational Quantum States

Every quantum circuit starts with all qubits in the the state $|0\rangle$. In order to prepare a quantum state $|\psi\rangle$ we need to prepare a `QuantumCircuit` that will modify the states of the qubits in order to get this specific state. The action of a circuit can always be represented as a unitiary operator.

\begin{align}
    |\psi\rangle &= \hat{U} |0 \ldots 0\rangle
\end{align}

For a parametric state the `QuantumCircuit` and therefore the unitary $U$ will depend on some parameters that we wirte as $\boldsymbol{\theta}$.

\begin{align}
    |\psi(\boldsymbol{\theta})\rangle &= \hat{U}(\boldsymbol{\theta}) |0 \ldots 0\rangle
\end{align}

We will see 2 ways to define Parametrized Quantum Circuits that represent Variational Quantum States. For the first method we only need the `QuantumCircuit` class from `qiskit.circuit`.

```python
from qiskit.circuit import QuantumCircuit
```

## Generating function
The easiest way to generate a parametrized `QuantumCircuit` is to implement a function that takes parameters as arguments and returns a `QuantumCircuit`. Here is such a function that generates a 2 qubits QuantumCircuit.

```python
def example_2qubits_2params_quantum_circuit(theta,phi):
    qc = QuantumCircuit(2)
    qc.ry(theta,0)
    qc.rz(phi,0)
    qc.cx(0,1)
    return qc
```

To visualize this circuit we first need to call the generating function with dummy argument values for it to return a circuit. We can draw the circuit. The `'mpl'` option draws the circuit in a fancy way using `matplotlib`. If you are experiencing problems, you can remove this option.

```python
varform_qc = example_2qubits_2params_quantum_circuit
qc = varform_qc(1,2)
qc.draw('mpl')
```

## Using qiskit parameter

The other way to generate a parametrized `QuantumCircuit` is to use the `Parameter` class in `qiskit`.

```python
from qiskit.circuit import Parameter
```

Here is the same circuit as before done with this method.

```python
a = Parameter('a')
b = Parameter('b')
varform_qc = QuantumCircuit(2)
varform_qc.ry(a,0)
varform_qc.rz(b,0)
varform_qc.cx(0,1)
```

    <qiskit.circuit.instructionset.InstructionSet at 0x13778e610>

Done this way the parametrized circuit can be drawn right away.

```python
varform_qc.draw('mpl')
```

To see what are the parameters of a parametrized `QuantumCircuit` you can use

```python
varform_qc.parameters
```

    {Parameter(a), Parameter(b)}

**Important** Beware that sometimes the parameters will not appear in the same order as you declared them!

To assign values to the different parameters we need to use the `QuantumCircuit.assign_paremeters()` method. This methods takes a `dict` as an argument containing the `Parameter`s and their `value`s.

```python
param_dict = {a : 1, b : 2}
qc = varform_qc.assign_parameters(param_dict)
qc.draw('mpl')
```

```python
param_dict = {a : 3, b : 4}
qc = varform_qc.assign_parameters(param_dict)
qc.draw('mpl')
```

If you want to provide the parameter values as a `list` or a `np.array` you can build the `dict` directly. Just make sure that the order you use in `param_values` corresponds to the other of `varform_qc.parameters`.

```python
param_values = [1, 2]
param_dict = dict(zip(varform_qc.parameters,param_values))
print(param_dict)
```

    {Parameter(a): 1, Parameter(b): 2}

## Varforms circuits for H2
Using the method of your choice, prepare 2 different 4-qubit `QuantumCircuit`s.
- The first should take 1 parameter to cover the real coefficients state sub space spanned by $|0101\rangle$ and $|1010\rangle$.
- The second should take 3 parameters to cover the real coefficients state sub space spanned by $|0101\rangle$, $|0110\rangle$, $|1001\rangle$ and $|1010\rangle$.

Revisit the presentation to find such circuits.

```python
varform_4qubits_1param = QuantumCircuit(4)
a = Parameter('a')
"""
Your code here
"""
varform_4qubits_1param.ry(a,1)
varform_4qubits_1param.x(0)
varform_4qubits_1param.cx(1,0)
varform_4qubits_1param.cx(0,2)
varform_4qubits_1param.cx(1,3)

varform_4qubits_1param.draw('mpl')
```

```python
varform_4qubits_3params = QuantumCircuit(4)
a = Parameter('a')
b = Parameter('b')
c = Parameter('c')
"""
Your code here
"""
varform_4qubits_3params.x(0)
varform_4qubits_3params.x(2)
varform_4qubits_3params.barrier(range(4))
varform_4qubits_3params.ry(a,1)
varform_4qubits_3params.cx(1,3)
varform_4qubits_3params.ry(b,1)
varform_4qubits_3params.ry(c,3)
varform_4qubits_3params.cx(3,2)
varform_4qubits_3params.cx(1,0)

varform_4qubits_3params.draw('mpl')
```

# Evaluator
The `Evaluator` is an object that will help us to evaluate the expectation value of a quantum operator (`LCPS`) on a specific variational form and backend. To initialize and `Evaluator` you should provide :

**Mandatory**
- A **variational form** that can create a `QuantumCircuit` given a set of `params`;
- A **backend** `qiskit.Backend` (a simulator or an actual device handle) on which to run the `QuantumCircuit`

**Optional**
- `execute_opt` is a `dict` containing the optionnal argument to pass to the `qiskit.execute` method (ex : `{'shots' : 1024}`.
- `measure_filter` a `qiskit.ignis...MeasurementFilter` that can be applied to the result of a circuit executation to mitigate readout errors.

The creation/usage of an `Evaluator` such as `BasicEvaluator` goes like this :

evaluator = BasicEvaluator(varform_qc,backend)
evaluator.set_linear_combinaison_pauli_string(operator_lcps)
expected_value = evaluator.eval(params)

First you initialize the evaluator.

Next, you provide the operator you want to evaluate using the `set_linear_combinaison_pauli_string(LCPS)` method.

Finally, you call the `eval(params)` method that will return the estimation of the operator's expected value. Mathematicaly, the use of this method corresponds to

\begin{align}
E(\boldsymbol{\theta}).
\end{align}

We will now go through the different pieces neccessary to complete the `Evaluator` class.

## Static methods
Being static, these method do not need an instance of a class to be used. They can be called directly from the class.

These methods are called before the first call to `eval(params)`. Most of these methods are implemented inside the abstract class `Evaluator` (except for `prepare_measurement_circuits_and_interpreters(LCPS)`)

### Pauli Based Measurements
We have seen that even if a quantum computer can only measure qubits in the Z-basis, the X and Y-basis are accessible if we *rotate* the quantum state before measuring.

Implement the `@staticmethod` : `pauli_string_based_measurement(PauliString)` in the `Evaluator` class in file `Evaluator.py` that returns a `QuantumCircuit` that measures each qubit in the basis given by the `PauliString`.

First we import the abstract class `Evaluator` and the `PauliString` class.

```python
from evaluator import Evaluator
from pauli_string import PauliString
```

Test your code with the next cell.

```python
%autoreload
pauli_string = PauliString.from_str('ZIXY')
measure_qc = Evaluator.pauli_string_based_measurement(pauli_string)
measure_qc.draw('mpl')
```

### Measurable eigenvalues

Implement the `@staticmethod` : `measurable_eigenvalues(PauliString)` in the `Evaluator` class in file `Evaluator.py` that returns a `np.array` that contains the eigenvalues of the measurable `PauliString` for each basis state. We noted this vector

\begin{align}
    \Lambda_q^{\hat{(\mathcal{P})}}.
\end{align}

Be mindful of the order of the basis state.

\begin{align}
    0000, 0001, 0010, \ldots, 1110, 1111
\end{align}

You can test your implementation on the `ZIXY` Pauli string.

```python
%autoreload
pauli_string = PauliString.from_str('ZIXY')
measurable_eigenvalues = Evaluator.measurable_eigenvalues(pauli_string)
print(measurable_eigenvalues)
```

    [ 1 -1 -1  1  1 -1 -1  1 -1  1  1 -1 -1  1  1 -1]

For the `PauliString` `'ZIXY'` (measurable `'ZIZZ'`) you should get the following eigenvalues :

[ 1 -1 -1  1  1 -1 -1  1 -1  1  1 -1 -1  1  1 -1]

### Measurement Circuits and Interpreters
The `prepare_measurement_circuits_and_interpreters(LCPS)` is specific to the sub-type of `Evaluator`. The two different types of `Evaluator`s considered in this workshop are :
- The `BasicEvaluator` will run a single `QuantumCircuit` for each `PauliString` present in the provided `LCPS`.
- The `BitwiseCommutingCliqueEvaluator` will exploit Bitwise Commuting Clique to combine the evaluation of Commuting `PauliStrin`s and reduce the number of different `QuantumCircuit` run for each evaluation.

Implement the `prepare_measurement_circuits_and_interpreters(LCPS)` method in the `BasicEvaluator` class in file `Evaluator.py`. This method should return 2 `list`. The first should contain one measurement `QuantumCircuit` for each `PauliString` in the `LCPS`. The second list should contain one `np.array` of the eigenvalues of the measurable `PauliString` for each basis state.

**Note** You can try to implement similar methods for the `BitwiseCommutingCliqueEvaluator`.

You can test your method on `2 ZIXY + 1 IXYZ`.

```python
from evaluator import BasicEvaluator
```

```python
%autoreload
lcps = 2*PauliString.from_str('ZIXY') + 1*PauliString.from_str('IXYZ')
measurement_circuits, interpreters = BasicEvaluator.prepare_measurement_circuits_and_interpreters(lcps)
```

You can visualize the interpreter and the measurement circuit for each term in the `LCPS` by using `i = 0` and `i = 1`.

```python
i = 1
print(interpreters[i])
measurement_circuits[i].draw('mpl')
```

The interpreters should be respectively :

[ 2 -2 -2  2  2 -2 -2  2 -2  2  2 -2 -2  2  2 -2]
[ 1 -1 -1  1 -1  1  1 -1  1 -1 -1  1 -1  1  1 -1]

### Set the LCPS
The method `set_linear_combinaison_pauli_string(LCPS)` is already implemented inside the abstract class `Evaluator`. Please take a look at it to notice that this method makes an immediate call to the `prepare_measurement_circuits_and_interpreters(LCPS)` method you have just implemented. The `measurement_circuits` and `interpreters` are also stored in attributes of the same name.

## Methods called inside `eval(params)`
Since we are entering the action of the `eval(params)` method we will need to instantiate an `Evaluator`. This will require a `backend`. We will use a local `qasm_simulator` for now, which is part of the `Aer` module. In the future, you can use a different `backend`. We will also soon need the `execute` method.

```python
from qiskit import Aer, execute
qasm_simulator = Aer.get_backend('qasm_simulator')
```

### Circuit preparation
The `prepare_eval_circuits(params)` will combine the variational form with these measurement `QuantumCircuit`s to form the complete circuit to be run. This method has 2 tasks :
- Assign the `params` to the variationnal form to get a `QuantumCircuit` that prepares the quantum state
- Combine this circuit with all the measurement circuits to return as many `QuantumCircuit` inside a `list`.

Implement this method inside the `Evaluator` class and test it here.

```python
%autoreload
lcps = 2*PauliString.from_str('ZXZX') + 1*PauliString.from_str('IIZZ')
varform = varform_4qubits_1param
backend = qasm_simulator
evaluator = BasicEvaluator(varform,backend)
evaluator.set_linear_combinaison_pauli_string(lcps)
params = [0,]
eval_circuits = evaluator.prepare_eval_circuits(params)
```

You can take a look at the `QuantumCircuit` for the first (`i=0`) and second (`i=1`) PauliString. What you should get is a circuit that begins with the state preparation circuit with the `params` applied to it followed by the measurement circuit.

```python
i = 0
eval_circuits[i].draw('mpl')
```

### Execution
The ultimate goal to the execution of a circuit is to get the number of times each basis state is measured. Let's execute our `eval_circuits`. We can run many `QuantumCircuit`s at the same time by placing them into a `list`, which they already are!

```python
execute_opts = {'shots' : 1024}
job = execute(eval_circuits, backend=qasm_simulator, **execute_opts)
result = job.result()
```

We can get the number of counts of each state for the execution of a given circuit with the follow lines. The counts are returned as a `dict`.

```python
i = 0
#i = 1
counts = result.get_counts(eval_circuits[i])
print(counts)
```

    {'0000': 266, '0001': 251, '0100': 262, '0101': 245}

If you `eval_circuits` are correct, you should get for, `i = 0` and `i = 1` respectively, something like this (exact value may vary since there is some randomness in the executation of a quantum circuit)

{'0000': 266, '0001': 262, '0100': 240, '0101': 256}
{'0101': 1024}

### counts2array

We will transform this `dict` into an array with the `counts2array` method.  Implement this method that will return the vector $N_q$. Be mindful of the order of the basis state.

\begin{align}
    0000, 0001, 0010, \ldots, 1110, 1111
\end{align}

**optional remark** While doing this will allow us to interpret the counts with a simple inner product, this implies creating an array of size $2^n$ where $n$ is the numbers of qubits. This might not be such a good idea for larger systems and the use of a `dict` might be more appropriate. Can you interpret the counts efficiently while keeping them in a `dict`?

```python
%autoreload
i = 0
counts = result.get_counts(eval_circuits[i])
evaluator.counts2array(counts)
```

    array([266, 251,   0,   0, 262, 245,   0,   0,   0,   0,   0,   0,   0,
             0,   0,   0])

For `i=0` in particular you should get something similar to:

array([228., 276.,   0.,   0., 269., 251.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.])

### Interpret counts
The action of interpreting the counts is actually the task of estimating the expectation value of a `PauliString` and multiplying by the coefficient associated with this `PauliString` in the LCPS. Implemented the `interpret_count_array` method that should return an array with the values for this expression.

\begin{align}
    h_i \langle \hat{\mathcal{P}}_i \rangle = \frac{h_i}{N_\text{tot}}\sum_{q} N_q \Lambda_q^{(\hat{\mathcal{P}}_i)}
\end{align}

```python
%autoreload
i = 1
counts_array = evaluator.counts2array(result.get_counts(eval_circuits[i]))
interpreter = evaluator.interpreters[i]
expected_value = evaluator.interpret_count_array(interpreter,counts_array)
print(expected_value)
```

    (-1+0j)

You should get something close to `0` for the first one and `-1` for the second.

### Evaluation
You have now all the pieces to complete the `eval(params)` method. This method should use all the methods you've implemented since the section *Methods called inside `eval(params)`* and then sum all the interpreted values. Mathematicaly, it should return the value of the expression

\begin{align}
    E(\boldsymbol{\theta}) = \sum_i h_i \langle\psi(\boldsymbol{\theta}) | \hat{\mathcal{P}}_i | \psi(\boldsymbol{\theta}) \rangle.
\end{align}

```python
%autoreload
lcps = 2*PauliString.from_str('ZXZX') + 1*PauliString.from_str('IIZZ')
varform = varform_4qubits_1param
backend = qasm_simulator
execute_opts = {'shots' : 1024}
evaluator = BasicEvaluator(varform, backend, execute_opts=execute_opts)
evaluator.set_linear_combinaison_pauli_string(lcps)
params = [0,]
expected_value = evaluator.eval(params)
print(expected_value)
```

    -1.0703125

Yes that's right, your code now returns an estimate of the expression

\begin{align}
    E(\theta) = \langle \psi(\theta) | \hat{\mathcal{H}} | \psi(\theta) \rangle.
\end{align}

for

\begin{align}
    \hat{\mathcal{H}} = 2\times \hat{Z}\hat{X}\hat{Z}\hat{X} + 1\times \hat{I}\hat{I}\hat{Z}\hat{Z}
\end{align}

and the varform `varform_4qubits_1param` for $\theta = 0$. The `evaluator.eval(params)` is now a method you can call like a function and it will return the energy $E(\theta)$.

Now comes the time to test this on the $\text{H}_2$ molecule Hamiltonian!

## The Hamiltonian evaluation test

We will now import the classes from the previous activity.

```python
from hamiltonian import MolecularFermionicHamiltonian
from mapping import JordanWigner
```

For ease of use we will import the integral values instead of using `pyscf`. We also import the Coulomb repulsion energy for later use. By now we are experts in building the Hamiltonian.

```python
with open('Integrals_sto-3g_H2_d_0.7350_no_spin.npz','rb') as f:
    out = np.load(f)
    h1_load_no_spin = out['h1']
    h2_load_no_spin = out['h2']
    energy_nuc =  out['energy_nuc']

molecular_hamiltonian = MolecularFermionicHamiltonian.from_integrals(h1_load_no_spin,h2_load_no_spin).include_spin()
```

We use the Jordan-Wigner mapping to the get the `LCPS` for the H2 molecule with `d=0.735`.

```python
%autoreload
mapping = JordanWigner()

lcps_h2 = mapping.fermionic_hamiltonian_to_linear_combinaison_pauli_string(molecular_hamiltonian).combine().apply_threshold().sort()
print(lcps_h2)
```

    15 pauli strings for 4 qubits (Real, Imaginary)
    IIII (-0.81055,+0.00000)
    IIIZ (+0.17218,+0.00000)
    IIZI (-0.22575,+0.00000)
    IIZZ (+0.12091,+0.00000)
    IZII (+0.17218,+0.00000)
    IZIZ (+0.16893,+0.00000)
    IZZI (+0.16615,+0.00000)
    ZIII (-0.22575,+0.00000)
    ZIIZ (+0.16615,+0.00000)
    ZIZI (+0.17464,+0.00000)
    ZZII (+0.12091,+0.00000)
    XXXX (+0.04523,+0.00000)
    XXYY (+0.04523,+0.00000)
    YYXX (+0.04523,+0.00000)
    YYYY (+0.04523,+0.00000)

We build an evaluator and feed it the `LCPS` of H2. And them we evaluate the energy. Use `params` in order that your `varform` prepares the state $|0101\rangle$.

```python
%autoreload
varform = varform_4qubits_1param
backend = qasm_simulator
execute_opts = {'shots' : 1024}
evaluator = BasicEvaluator(varform,backend,execute_opts = execute_opts)
evaluator.set_linear_combinaison_pauli_string(lcps_h2)
params = [0,]
expected_value = evaluator.eval(params)
print(expected_value)
```

    -1.8370563365153778

If your `varform` prepares the state $|0101\rangle$, you should get something around `-1.83`. This energy is already close to the ground state energy because the ground state is close to $|0101\rangle$, but still it's not the ground state. We need to find the `params` that will minimise the energy.

\begin{align}
    E_0 = \min_{\boldsymbol{\theta}} E(\boldsymbol{\theta})
\end{align}

# Solver

In a final step we need to implement a solver that will try to find the minimal energy. We will implement 2 solvers. The second is optional.
- First the one using the VQE algo in conjunction with a minimizer to try to minimize `evaluator.eval(params)`.
- Next we will make use of the `to_matrix()` method you implemented in the previous activity to find the exact value/solution.

## VQE Solver

Like any minimzation process this solver will need a couple of ingredients :
- A function to minimize, we will provide this with the evaluator
- A minimizer, an algorithm that generaly takes in a function and a set of starting parameters and returns the best guess for the optimal parameters that correspond to the minimal value of the function to minimize.
- A set of starting parameters.

### Minimizer

A minimizer that works OK for the VQE algorithme is the  Sequential Least SQuares Programming (SLSQP) algorithm. It's available in the `minimize` sub-module of [scipy](https://docs.scipy.org/doc/scipy/reference/optimize.minimize-slsqp.html).

```python
from scipy.optimize import minimize
```

We will make a Lambda function with the minimizer so we can set all sorts of parameter before feeding it to the solver.

```python
minimizer = lambda fct, start_param_values : minimize(
    fct,
    start_param_values,
    method = 'SLSQP',
    options = {'maxiter' : 5,'eps' : 1e-1, 'ftol' : 1e-4, 'disp' : True, 'iprint' : 2})
```

The `minimizer` now takes only 2 arguments : the function and the starting parameters values. We also specify some options :
- A small value for the maximum number of iteration. You will find that running the VQE algorithm is expensive because of the `evaluator.eval(params)` method. Either it's long to simulate on `qasm_simulator` or because it's running on an actual quantum computer.
- A `eps` of `0.1`. This is the size of the step the algorithm is going to change the values of the parameters to try to estimate the slope of the function. By the way, a lot of minimizing algorithms use the slope of the function to know in which direction is the minimum. Since our parameters are all angles in radians a value of 0.1 seems reasonnable. Play with this value if you like.
- A `ftol` value of `1e-4`. This is the goal for the precision of the value of the minimum value. The chemical accuracy is around 1 milli-Hartree.
- We set `iprint` to `2` so see what is going on. For your final implementation you can set this to `0`.

Before implementing the `VQESolver` let's try this minimizer! The function is `evaluator.eval` and we start with a parameter of `0`. This will take a while.

```python
minimization_result = minimizer(evaluator.eval,[0,])
```

      NIT    FC           OBJFUN            GNORM
        1     3    -1.834406E+00     2.276211E-01
        2     6    -1.857755E+00     4.369813E-02
        3    11    -1.860118E+00     1.790857E-02
    Optimization terminated successfully.    (Exit mode 0)
                Current function value: -1.8601177724457674
                Iterations: 3
                Function evaluations: 11
                Gradient evaluations: 3

In the end you should get an minimal energy around `1.86` Hartree. Which is a bit smaller then what we had before minimizing. You can explore the `minimization_result` to retreive this value but also the set of optimal parameters.

```python
opt_params = minimization_result.x
opt_value = minimization_result.fun
print(opt_params)
print(opt_value)
```

    [-0.22882531]
    -1.8601177724457674

### VQE Solver

Now you should be in good position to implement the `lowest_eig_value(lcps)` of the `VQESolver` class inside the `Solve.py` file. Test your method here.

```python
from solver import VQESolver
```

```python
%autoreload
vqe_solver = VQESolver(evaluator, minimizer, [0,], name = 'vqe_solver')
opt_value, opt_params = vqe_solver.lowest_eig_value(lcps_h2)
```

      NIT    FC           OBJFUN            GNORM
        1     3    -1.839000E+00     2.553151E-01
        2     6    -1.865535E+00     6.395731E-02
    Optimization terminated successfully.    (Exit mode 0)
                Current function value: -1.865591974724343
                Iterations: 2
                Function evaluations: 10
                Gradient evaluations: 2

The is only one thing missing to have the complete molecular energy : the Coulomb repulsion energy from the nucleus. This value was loaded when we imported the integrals. Let's add it to the electronic energy.

```python
print('Ground state position estimate (vqe) : ', opt_params)
print('Ground state energy estimate (electronic, vqe) : ', opt_value)
print('Ground state energy estimate (molecular, vqe) : ', opt_value + energy_nuc)
```

    Ground state position estimate (vqe) :  [-0.25526396]
    Ground state energy estimate (electronic, vqe) :  -1.865591974724343
    Ground state energy estimate (molecular, vqe) :  -1.1456229802753635

### The Eigenstate

What is the eigenstate? We can partially find out by using the `varform` with the parameters we have found and measure everything in the Z basis.

```python
eigenstate_qc = varform.copy()
eigenstate_qc.measure_all()

param_dict = dict(zip(eigenstate_qc.parameters,opt_params))
eigenstate_qc = eigenstate_qc.assign_parameters(param_dict)

eigenstate_qc.draw('mpl')
```

We now execute this circuit.

```python
execute_opts = {'shots' : 1024}
job = execute(eigenstate_qc,backend=qasm_simulator,**execute_opts)
result = job.result()
counts = result.get_counts(eigenstate_qc)
```

We will use the `plot_histogram` method from `qiskit.visualization` that takes the counts `dict` as an input.

```python
from qiskit.visualization import plot_histogram
```

```python
plot_histogram(counts)
```

```python
print(f"|a_0101| ~ {np.sqrt(counts['0101']/1024)}")
print(f"|a_1010| ~ {np.sqrt(counts['1010']/1024)}")
```

    |a_0101| ~ 0.9936320684740404
    |a_1010| ~ 0.11267347735824966

We see that the found solution is mostly the state $|0101\rangle$ which is the Hartree-Fock solution when the 2-body Hamiltonian is not present. Adding this 2-body physics, shifts the energy down a bit by introducing a small contribution of $|1010\rangle$. The actual statevector has a `-` sign between these two states.

\begin{align}
\alpha_{0101}|0101\rangle - \alpha_{1010}|1010\rangle
\end{align}

But this is not something we can know from this. Fortunatly, H2 is a small system which can be solved exactly and we can find out this phase.

## Exact Solver (optional)

If you want to compare the value you get with the VQE algorithm it would be nice to have the exact value. If you were able to implement the `to_matrix()` method for `PauliString` and `LinearCombinaisonPauliString` then you can find the exact value of the ground state. All you need is to diagonalise the matrix reprensenting the whole Hamiltonian and find the lowest eigenvalue! Obviously this will not be possible to do for very large systems.

```python
hamiltonian_matrix_h2 = lcps_h2.to_matrix()
eig_values, eig_vectors = np.linalg.eigh(hamiltonian_matrix_h2)
eig_order = np.argsort(eig_values)
eig_values = eig_values[eig_order]
eig_vectors = eig_vectors[:,eig_order]
ground_state_value, ground_state_vector = eig_values[0], eig_vectors[:,0]
print('Ground state vector (exact) : \n', ground_state_vector)
print('Ground state energy (electronic, exact) : ', ground_state_value)
print('Ground state energy (molecular, exact) : ', ground_state_value + energy_nuc)
```

    Ground state vector (exact) :
     [-0.        -0.j -0.        -0.j -0.        -0.j -0.        -0.j
     -0.        -0.j -0.9937604 -0.j  0.        +0.j  0.        +0.j
      0.        +0.j  0.        +0.j  0.11153594+0.j  0.        +0.j
      0.        +0.j  0.        +0.j  0.        +0.j  0.        +0.j]
    Ground state energy (electronic, exact) :  -1.8572750302023788
    Ground state energy (molecular, exact) :  -1.137306035753399

Now you can complete the `ExactSolver` in the `Solver.py` file.

```python
from solver import ExactSolver
```

```python
%autoreload
exact_solver = ExactSolver()
ground_state_value, ground_state_vector = exact_solver.lowest_eig_value(lcps_h2)
print('Ground state vector (exact) : ', ground_state_vector)
print('Ground state energy (electronic, exact) : ', ground_state_value)
print('Ground state energy (molecular, exact) : ', ground_state_value + energy_nuc)
```

    Ground state vector (exact) :  [-0.        -0.j -0.        -0.j -0.        -0.j -0.        -0.j
     -0.        -0.j -0.9937604 -0.j  0.        +0.j  0.        +0.j
      0.        +0.j  0.        +0.j  0.11153594+0.j  0.        +0.j
      0.        +0.j  0.        +0.j  0.        +0.j  0.        +0.j]
    Ground state energy (electronic, exact) :  -1.8572750302023788
    Ground state energy (molecular, exact) :  -1.137306035753399

What are the two basis state involved in the ground state? Let's plot the state vector using `matplotlib`.

```python
import matplotlib.pyplot as plt
```

```python
fig,ax = plt.subplots(1,1)
i_max = np.argmax(np.abs(ground_state_vector))
state = ground_state_vector * np.sign(ground_state_vector[i_max])
ax.bar(range(len(state)), np.abs(state), color=(np.real(state) > 0).choose(['r','b']))
plt.xticks(range(len(state)),[f"{i:04b}" for i in range(len(state))], size='small',rotation=60);
```

# What's next?

Now that you can find the ground state for a specific H2 molecule configuration (`d = 0.735`), you should be able to do that for many configurations, say `d = 0.2` to `2.5`. Doing that will enable you to plot the so-called dissociation curve : energy vs distance. Do not forget to include the Coulomb repulsion energy of the nucleus!

You could also run your algorithm on a noisy backend, either a noisy simulator or a real quantum computer. You've already seen on day 1 how to set/get a noisy backend. You'll see that noise messes things up pretty bad.

Running on real machine will introduce the problem of the qubit layout. You might want to change the `initial_layout` in the `execute_opts` so that your `varform` is not applying CNOT gates between qubits that are not connected. You know this needs to insert SWAP gate and this introduce more noise. Also covered in day 1.

To limit the effect of readout noise, you could add a `measure_filter` to your `evaluator`, so that each time you execute the `eval_circuits` you apply the filter to the results. Also covered in day 1.

Implement the simulatneous evaluation for bitwise commuting cliques or even for general commuting cliques.

Notebook by **Maxime Dion** <maxime.dion@usherbrooke.ca>

For the QSciTech-QuantumBC virtual workshop on gate-based quantum computing

### Plot the H2 Dissociation Curve

```python
from pyscf import gto
```

```python
%autoreload
n = 50
distances = np.linspace(0.3, 2.5, n)
gs_energies_exact = np.zeros(n)
gs_energies_vqe = np.zeros(n)
energy_nuc = np.zeros(n)

# define mapping
mapping = JordanWigner()

# define minimizer
minimizer = lambda fct, start_param_values : minimize(
    fct,
    start_param_values,
    method = 'SLSQP',
    options = {'maxiter' : 5,'eps' : 1e-1, 'ftol' : 1e-4})

# instantiate an exact solver for comparison
exact_solver = ExactSolver()

# VQE setup
vqe_evaluator = BasicEvaluator(varform_4qubits_1param, qasm_simulator, execute_opts={'shots' : 1024})
vqe_solver = VQESolver(evaluator, minimizer, [0,], name = 'vqe_solver')

# try a range of internuclear distances
for i, distance in enumerate(distances): #units in AA
    print('Trying Distance '+str(i+1), end="\r")

    # build the molecule and basis functions
    mol = gto.M(
        atom = [['H', (0,0,-distance/2)], ['H', (0,0,distance/2)]],
        basis = 'sto-3g'
        )

    # build the molecular Hamiltonian
    molecular_hamiltonian = MolecularFermionicHamiltonian.from_pyscf_mol(mol).include_spin()

    # map the Hamiltonian to a LCPS
    lcps_h2 = mapping.fermionic_hamiltonian_to_linear_combinaison_pauli_string(molecular_hamiltonian).combine().apply_threshold().sort()

    # store the nuclear energy
    energy_nuc[i] = mol.energy_nuc()

    # diagonalize the Hamiltonian to get energies
    Eh2_exact, _ = exact_solver.lowest_eig_value(lcps_h2)
    gs_energies_exact[i] = Eh2_exact + energy_nuc[i]

    # get the vqe energy
    Eh2_vqe, _ = vqe_solver.lowest_eig_value(lcps_h2)
    gs_energies_vqe[i] = Eh2_vqe + energy_nuc[i]

print("Done!                   ", end="\r")
```

    Done!

```python
# plot dissociation curve of H2
fig, ax = plt.subplots(1, 1, figsize=(10,8))
ax.plot(distances, gs_energies_exact, c='tab:red', label='Exact', linewidth=5)
ax.plot(distances, gs_energies_vqe, '.', c='tab:blue', label='VQE', ms=20)
ax.set_xlabel(r'Internuclear Distance / $\AA$', fontsize=20)
ax.set_ylabel('Energy / $E_h$', fontsize=20)
ax.set_title('Dissociation Curve of H2', fontsize=28)
ax.legend()
fig.savefig('H2_dissociation.png')
plt.show()
```

```python
# save these results
with open('h2_dissociation.npz','wb') as f:
    np.savez(f, atom='H2', basis=mol.basis, distances=distances, energy_nuc=energy_nuc, gs_exact=gs_energies_exact,
             gs_vqe=gs_energies_vqe, varform='varform_4qubits_1param', backend='qasm_simulator', execute_opts=execute_opts,
             mapping='Jordan Wigner', initial_params=[0,], minimizer='SLSQP', minimizer_options={'maxiter' : 5,'eps' : 1e-1, 'ftol' : 1e-4})
```

```python
# to reload any data...
with open('h2_dissociation.npz','rb') as f:
    out = np.load(f, allow_pickle=True)
    varform_load = out['varform']
    basis_load = out['basis']
    energy_nuc_load = out['energy_nuc']
    gs_exact_load = out['gs_exact']
    gs_vqe_load = out['gs_vqe']
    execute_opts_load = out['execute_opts']
```

# Now Let's Add A Realistic Noise Model to our Simulator

```python
from qiskit import IBMQ
from qiskit.providers.aer.noise import NoiseModel

# IBMQ.save_account(TOKEN)
IBMQ.load_account()
IBMQ.providers()

#provider = IBMQ.get_provider(hub='ibm-q-education')

provider = IBMQ.get_provider(hub='ibm-q-education', group='qscitech-quantum', project='qc-bc-workshop')
provider2 = IBMQ.get_provider(hub='ibm-q', group='open', project='main')
```

    /Users/bhenders/opt/miniconda3/envs/qiskit/lib/python3.8/site-packages/qiskit/providers/ibmq/ibmqfactory.py:192: UserWarning: Timestamps in IBMQ backend properties, jobs, and job results are all now in local time instead of UTC.
      warnings.warn('Timestamps in IBMQ backend properties, jobs, and job results '
    ibmqfactory.load_account:WARNING:2021-02-01 14:15:45,480: Credentials are already in use. The existing account in the session will be replaced.

## Let's Play Around With a Few Backends with different Topologies

```python
bogota = provider.get_backend('ibmq_bogota')
# santiago = provider.get_backend('ibmq_santiago')
casablanca = provider.get_backend('ibmq_casablanca')
rome = provider.get_backend('ibmq_rome')
qasm_simulator = Aer.get_backend('qasm_simulator')
valencia = provider2.get_backend('ibmq_valencia')
melbourne = provider2.get_backend('ibmq_16_melbourne')
```

```python
# Bogota
bogota_prop = bogota.properties()
bogota_conf = bogota.configuration()
bogota_nm = NoiseModel.from_backend(bogota_prop)

# Casablanca
casablanca_conf = casablanca.configuration()
casablanca_prop = casablanca.properties()
casablanca_nm = NoiseModel.from_backend(bogota_prop)

# Valencia
valencia_conf = valencia.configuration()
valencia_prop = valencia.properties()
valencia_nm = NoiseModel.from_backend(bogota_prop)

# Melbourne
melbourne_conf = melbourne.configuration()
melbourne_prop = melbourne.properties()
melbourne_nm = NoiseModel.from_backend(bogota_prop)
```

```python
execute_opts = {'shots' : 1024,
                'noise_model': bogota_nm,
                'coupling_map':bogota_conf.coupling_map,
                'basis_gates':bogota_conf.basis_gates}
evaluator = BasicEvaluator(varform,backend,execute_opts = execute_opts)
vqe_solver = VQESolver(evaluator, minimizer, [0,], name = 'vqe_solver')
opt_energy, opt_params = vqe_solver.lowest_eig_value(lcps_h2)
```

      NIT    FC           OBJFUN            GNORM
        1     3    -1.644299E+00     4.420509E-02
    Optimization terminated successfully.    (Exit mode 0)
                Current function value: -1.6534026525155174
                Iterations: 1
                Function evaluations: 9
                Gradient evaluations: 1

```python
execute_opts = {'shots' : 1024}
evaluator = BasicEvaluator(varform, bogota, execute_opts=execute_opts)
vqe_solver = VQESolver(evaluator, minimizer, [0,], name = 'vqe_solver_bogota')
opt_energy, opt_params = vqe_solver.lowest_eig_value(lcps_h2)
```

      NIT    FC           OBJFUN            GNORM
        1     3    -1.615201E+00     3.814424E-01
        2    10    -1.619136E+00     8.165022E-02
    Optimization terminated successfully.    (Exit mode 0)
                Current function value: -1.6191355224178043
                Iterations: 2
                Function evaluations: 10
                Gradient evaluations: 2

Notice that the ground state energy of an $H_2$ molecule is found to be -1.653 $E_h$ when our circuit runs on a noisy backend, with the same qubit coupling map as the Bogota device. When we run on the actual Bogota backend, we obtain an even worse result of -1.619 $E_h$. We compare this to -1.866 $E_h$ when running on an ideal simulator with only statistical noise contributing to potential error. We can try to mitigate some of this discrepancy by applying a `MeasurementFilter` to our circuit measurements.

**Plan of Attack:**
1. Decide on an optimal qubit layout (or 2 good ones) to minimize extra CX gates and error-prone U2 gates.
2. Generate **measurement calibration circuits** using these layouts.
3. Add a measurement filter to the VQE evaluator
4. Compare to un-filtered results

Note that our circuit uses 4 qubits. Perhaps one of the biggest optimizations we could do would be using parity mapping to reduce qubit requirements.

### Qubit Mapping

Keep in mind our variational circuit:

```python
qc = varform_4qubits_1param.assign_parameters({a: 1})
qc.draw()
```

Take a look at the coupling on Bogota, the actual machine we hope to use. Then examine the error rates for the CX and single qubit unitaries on the Bogota machine to get a sense of how we might better map our problem to this device.

```python
bogota_conf.coupling_map
```

    [[0, 1], [1, 0], [1, 2], [2, 1], [2, 3], [3, 2], [3, 4], [4, 3]]

```python
# Print CNOT error from Bogota calibration data
cx_errors = list(map(lambda cm: bogota_prop.gate_error("cx", cm), bogota_conf.coupling_map))
for i in range(len(bogota_conf.coupling_map)):
    print(f'      -> qubits {bogota_conf.coupling_map[i]} CNOT error: {cx_errors[i]}')
```

          -> qubits [0, 1] CNOT error: 0.02502260182900265
          -> qubits [1, 0] CNOT error: 0.02502260182900265
          -> qubits [1, 2] CNOT error: 0.010158268780223023
          -> qubits [2, 1] CNOT error: 0.010158268780223023
          -> qubits [2, 3] CNOT error: 0.014415524414420677
          -> qubits [3, 2] CNOT error: 0.014415524414420677
          -> qubits [3, 4] CNOT error: 0.010503141811223582
          -> qubits [4, 3] CNOT error: 0.010503141811223582

* CNOT gates between [0,1] and between [2,3] seem to have the largest error. Can we avoid these?

```python
# Print U2 error from Bogota calibration data
u2_errors = list(map(lambda q: bogota_prop.gate_error("sx", q), range(bogota_conf.num_qubits)))
for i in range(bogota_conf.num_qubits):
    print(f'      -> qubits {i} U2 error: {u2_errors[i]}')
```

          -> qubits 0 U2 error: 0.00031209152498965555
          -> qubits 1 U2 error: 0.00029958716199301446
          -> qubits 2 U2 error: 0.00017693377775244873
          -> qubits 3 U2 error: 0.0004023787341145875
          -> qubits 4 U2 error: 0.00016725621608793646

* Qubit 3 seems to have the largest error. Can we avoid using it?

Let's experiment with several different qubit layouts to see if we can reduce the number of CX gates on problematic pairs and U2 gates on problematic qubits.

```python
# all of these have optimal number of CNOTS (3)
layout = [2,3,1,4]  # 1
layout = [1,2,0,3]  # 2
layout = [3,2,4,1]  # 3 Looks the most promising
layout = [2,1,3,0]  # 4

# These are the equivalent topologies on Valencia
layout_valencia = [1,3,2,4]
layout_valencia = [1,3,4,2]
layout_valencia = [3,1,4,2]
layout_valencia = [3,1,2,4]
layout_valencia = [3,1,4,0]
layout_valencia = [1,3,4,0]

qc_l1 = transpile(qc,
                  coupling_map=bogota_conf.coupling_map,
                  basis_gates=bogota_conf.basis_gates,
                  initial_layout=layout,
                  optimization_level=1)
qc_l1.draw()
```

```python
print(f'Original circuit depth: {qc.depth()} - Transpiled circuit depth: {qc_l1.depth()}')
```

    Original circuit depth: 3 - Transpiled circuit depth: 6

**Summary of Findings**:

* **Layout 1**: depth 6 with opt level 1 (no improvement for higher)
    - downside: uses cx between 0, 1, which has highest error rate, plus 4 U2s on q3, which has highest error rate. one cx on 2,3
* **Layout 2**: depth 6 with opt level 1 (no improvement for higher)
    - downside: uses cx between 0, 1, which has highest error rate, one cx on 2,3
* **Layout 3**: depth 6 with opt level 1 (no improvement for higher)
    - downside: One cx between 2,3
* **Layout 4**: depth 6 with opt level 1 (no improvement for higher)
    - downside: uses cx between 0, 1, which has highest error rate, plus one on 2,3.

**Conclusion**: Layout 3 is probably optimal.

## Now Create a Measurement Filter

```python
from qiskit.circuit import QuantumRegister
from qiskit.ignis.mitigation.measurement import complete_meas_cal

# Generate the calibration circuits for the 3 qubits we measure
qr = QuantumRegister(4)

# we need our measurement filter to handle 4 qubits
qubit_list = [0,1,2,3]

# meas_calibs is a list containing 2^n circuits, one for each state.
meas_calibs, state_labels = complete_meas_cal(qubit_list=qubit_list, qr=qr, circlabel='mcal')

print(f'Number of circuits: {len(meas_calibs)}')
meas_calibs[1].draw()
```

```python
# We need the filter to correspond to the layout we are using
calibration_layout = [3,2,4,1]
result = execute(meas_calibs,
                 qasm_simulator,
                 shots=8192,
                 noise_model=bogota_nm,
                 coupling_map=bogota_conf.coupling_map,
                 basis_gates=bogota_conf.basis_gates,
                 initial_layout=calibration_layout).result()
```

```python
from qiskit.visualization import plot_histogram

# For example, plot histogram for circuit corresponding to state '001' (index 1)
plot_histogram(result.get_counts(meas_calibs[5]))
```

```python
from qiskit.ignis.mitigation.measurement import CompleteMeasFitter

# Initialize the measurement correction fitter for a full calibration
meas_fitter = CompleteMeasFitter(result, state_labels)

# Get the filter object
meas_filter = meas_fitter.filter

fig, ax = plt.subplots(1,1, figsize=(10,8))
meas_fitter.plot_calibration(ax=ax)
fig.savefig('images/4_qubit_measurement_filter.svg')
```

### Bogota Noise Model But With all Default Qubit Layout and No Measurement Filtersavefig

```python
%autoreload
execute_opts = {'shots' : 1024,
                'noise_model': bogota_nm,
                'coupling_map':bogota_conf.coupling_map,
                'basis_gates':bogota_conf.basis_gates,
                }
evaluator = BasicEvaluator(varform_4qubits_1param, qasm_simulator,execute_opts=execute_opts, measure_filter=None)
vqe_solver = VQESolver(evaluator, minimizer, [0,], name='vqe_solver')
energy_default, opt_params_default = vqe_solver.lowest_eig_value(lcps_h2)
```

      NIT    FC           OBJFUN            GNORM
        1     3    -1.660741E+00     1.105373E-01
    Optimization terminated successfully.    (Exit mode 0)
                Current function value: -1.6647674403625325
                Iterations: 1
                Function evaluations: 10
                Gradient evaluations: 1

### Bogota Noise Model With An Improved Qubit Layout but no Measurement Filter

```python
%autoreload
execute_opts = {'shots' : 1024,
                'noise_model': bogota_nm,
                'coupling_map':bogota_conf.coupling_map,
                'basis_gates':bogota_conf.basis_gates,
                'initial_layout': [3,2,4,1]
                }
evaluator = BasicEvaluator(varform_4qubits_1param, qasm_simulator,execute_opts=execute_opts, measure_filter=None)
vqe_solver = VQESolver(evaluator, minimizer, [0,], name='vqe_solver')
energy_layout, opt_params_layout = vqe_solver.lowest_eig_value(lcps_h2)
```

      NIT    FC           OBJFUN            GNORM
        1     3    -1.684631E+00     2.120353E-01
        2     8    -1.689694E+00     2.218513E-01
        3    14    -1.697248E+00     3.986125E-02
        4    18    -1.704745E+00     1.969790E-01
    Optimization terminated successfully.    (Exit mode 0)
                Current function value: -1.6806716594964146
                Iterations: 4
                Function evaluations: 29
                Gradient evaluations: 4

### Bogota Noise Model With An Improved Qubit Layout and Measurement Filter

```python
%autoreload
execute_opts = {'shots' : 1024,
                'noise_model': bogota_nm,
                'coupling_map':bogota_conf.coupling_map,
                'basis_gates':bogota_conf.basis_gates,
                'initial_layout': [3,2,4,1]
                }
evaluator = BasicEvaluator(varform_4qubits_1param, qasm_simulator,execute_opts=execute_opts, measure_filter=meas_filter)
vqe_solver = VQESolver(evaluator, minimizer, [0,], name='vqe_solver')
energy_layout_meas, opt_params_layout_meas = vqe_solver.lowest_eig_value(lcps_h2)
```

      NIT    FC           OBJFUN            GNORM
        1     3    -1.789338E+00     7.476305E-02
        2     6    -1.821437E+00     1.549079E-01
        3    10    -1.825480E+00     8.058220E-02
    Optimization terminated successfully.    (Exit mode 0)
                Current function value: -1.8349246159398094
                Iterations: 3
                Function evaluations: 15
                Gradient evaluations: 3

### Analysis

**Running with a Simulated Noise Model and 1024 Shots**

Calculated electronic energies at $0.735 Angstrom$

| Method | Shots | Backend | Noise Model | Measurement Filter | Energy ($E_h$) | % Error |
|------|-------|------|-----|-----|-----|-------|
| Exact | N/A | N/A | N/A | N/A | -1.857275 | 0 |
| VQE | 1024 | Simulator | None | No | -1.86559 | -0.45 |
| VQE | 1024 | Simulator | Bogota | No | -1.66074 | 10.58 |
| VQE (layout optimized) | 1024 | Simulator | Bogota | No | -1.68067 | 9.51 |
| VQE (layout optimized) | 1024 | Simulator | Bogota | Yes | -1.83492 | 1.20 |
| VQE | 1024 | Bogota | N/A | No | -1.61914 | 12.82 |

With no optimizations, the calculated ground state energy is off by approximately 10.6% relative to the "exact" solution. Adding an improved qubit layout, we reduce this error to between 9 and 10%. Applying a measurement filter on top of this, we obtain a final error of about 1.2%.

Measurement errors therefore seem to be the most significant source of error for such a short circuit, especially once the number of required CX gates is reduced to 3. This circuit is short enough that decoherence does not play a very large role.

**Running on Bogota and 1024 Shots**

???

**Next Steps:**
We should examine how this error contributes to the dissociation curve, and whether it significantly alters the predicted equilibrium distance for $H_2$

```python

```

```python

```


```python
from google.colab import drive
drive.mount('/content/drive')
```

    Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3aietf%3awg%3aoauth%3a2.0%3aoob&response_type=code&scope=email%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdocs.test%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive.photos.readonly%20https%3a%2f%2fwww.googleapis.com%2fauth%2fpeopleapi.readonly

    Enter your authorization code:
    ··········
    Mounted at /content/drive

# Neuromatch Academy: Week 2, Day 5, Tutorial 1
# Learning to Predict

__Content creators:__ Marcelo Mattar and Eric DeWitt with help from Matt Krause

__Content reviewers:__ Byron Galbraith and Michael Waskom

---

# Tutorial objectives

In this tutorial, we will learn how to estimate state-value functions in a classical conditioning paradigm using Temporal Difference (TD) learning and examine TD-errors at the presentation of the conditioned and unconditioned stimulus (CS and US) under different CS-US contingencies. These exercises will provide you with an understanding of both how reward prediction errors (RPEs) behave in classical conditioning and what we should expect to see if Dopamine represents a "canonical" model-free RPE.

At the end of this tutorial:
* You will learn to use the standard tapped delay line conditioning model
* You will understand how RPEs move to CS
* You will understand how variability in reward size effects RPEs
* You will understand how differences in US-CS timing effect RPEs

```python
# Imports
import numpy as np
import matplotlib.pyplot as plt
```

```python
#@title Figure settings
import ipywidgets as widgets       # interactive display
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")
```

```python
# @title Helper functions
from matplotlib import ticker

def plot_value_function(V, ax=None, show=True):
  """Plot V(s), the value function"""
  if not ax:
    fig, ax = plt.subplots()

  ax.stem(V, use_line_collection=True)
  ax.set_ylabel('Value')
  ax.set_xlabel('State')
  ax.set_title("Value function: $V(s)$")

  if show:
    plt.show()

def plot_tde_trace(TDE, ax=None, show=True, skip=400):
  """Plot the TD Error across trials"""
  if not ax:
    fig, ax = plt.subplots()

  indx = np.arange(0, TDE.shape[1], skip)
  im = ax.imshow(TDE[:,indx])
  positions = ax.get_xticks()
  # Avoid warning when setting string tick labels
  ax.xaxis.set_major_locator(ticker.FixedLocator(positions))
  ax.set_xticklabels([f"{int(skip * x)}" for x in positions])
  ax.set_title('TD-error over learning')
  ax.set_ylabel('State')
  ax.set_xlabel('Iterations')
  ax.figure.colorbar(im)
  if show:
    plt.show()

def learning_summary_plot(V, TDE):
  """Summary plot for Ex1"""
  fig, (ax1, ax2) = plt.subplots(nrows = 2, gridspec_kw={'height_ratios': [1, 2]})

  plot_value_function(V, ax=ax1, show=False)
  plot_tde_trace(TDE, ax=ax2, show=False)
  plt.tight_layout()

def reward_guesser_title_hint(r1, r2):
  """"Provide a mildly obfuscated hint for a demo."""
  if (r1==14 and r2==6) or (r1==6 and r2==14):
    return "Technically correct...(the best kind of correct)"

  if  ~(~(r1+r2) ^ 11) - 1 == (6 | 24): # Don't spoil the fun :-)
    return "Congratulations! You solved it!"

  return "Keep trying...."

#@title Default title text
class ClassicalConditioning:

    def __init__(self, n_steps, reward_magnitude, reward_time):

        # Task variables
        self.n_steps = n_steps
        self.n_actions = 0
        self.cs_time = int(n_steps/4) - 1

        # Reward variables
        self.reward_state = [0,0]
        self.reward_magnitude = None
        self.reward_probability = None
        self.reward_time = None

        self.set_reward(reward_magnitude, reward_time)

        # Time step at which the conditioned stimulus is presented

        # Create a state dictionary
        self._create_state_dictionary()

    def set_reward(self, reward_magnitude, reward_time):

        """
        Determine reward state and magnitude of reward
        """
        if reward_time >= self.n_steps - self.cs_time:
            self.reward_magnitude = 0

        else:
            self.reward_magnitude = reward_magnitude
            self.reward_state = [1, reward_time]

    def get_outcome(self, current_state):

        """
        Determine next state and reward
        """
        # Update state
        if current_state < self.n_steps - 1:
            next_state = current_state + 1
        else:
            next_state = 0

        # Check for reward
        if self.reward_state == self.state_dict[current_state]:
            reward = self.reward_magnitude
        else:
            reward = 0

        return next_state, reward

    def _create_state_dictionary(self):

        """
        This dictionary maps number of time steps/ state identities
        in each episode to some useful state attributes:

        state      - 0 1 2 3 4 5 (cs) 6 7 8 9 10 11 12 ...
        is_delay   - 0 0 0 0 0 0 (cs) 1 1 1 1  1  1  1 ...
        t_in_delay - 0 0 0 0 0 0 (cs) 1 2 3 4  5  6  7 ...
        """
        d = 0

        self.state_dict = {}
        for s in range(self.n_steps):
            if s <= self.cs_time:
                self.state_dict[s] = [0,0]
            else:
                d += 1 # Time in delay
                self.state_dict[s] = [1,d]

class MultiRewardCC(ClassicalConditioning):
  """Classical conditioning paradigm, except that one randomly selected reward,
    magnitude, from a list, is delivered of a single fixed reward."""
  def __init__(self, n_steps, reward_magnitudes, reward_time=None):
    """"Build a multi-reward classical conditioning environment
      Args:
        - nsteps: Maximum number of steps
        - reward_magnitudes: LIST of possible reward magnitudes.
        - reward_time: Single fixed reward time
      Uses numpy global random state.
      """
    super().__init__(n_steps, 1, reward_time)
    self.reward_magnitudes = reward_magnitudes

  def get_outcome(self, current_state):
    next_state, reward = super().get_outcome(current_state)
    if reward:
      reward=np.random.choice(self.reward_magnitudes)
    return next_state, reward

class ProbabilisticCC(ClassicalConditioning):
  """Classical conditioning paradigm, except that rewards are stochastically omitted."""
  def __init__(self, n_steps, reward_magnitude, reward_time=None, p_reward=0.75):
    """"Build a multi-reward classical conditioning environment
      Args:
        - nsteps: Maximum number of steps
        - reward_magnitudes: Reward magnitudes.
        - reward_time: Single fixed reward time.
        - p_reward: probability that reward is actually delivered in rewarding state
      Uses numpy global random state.
      """
    super().__init__(n_steps, reward_magnitude, reward_time)
    self.p_reward = p_reward

  def get_outcome(self, current_state):
    next_state, reward = super().get_outcome(current_state)
    if reward:
      reward*= int(np.random.uniform(size=1)[0] < self.p_reward)
    return next_state, reward

```

---
# Section 1: TD-learning

```python
#@title Video 1: Introduction
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="YoNbc9M92YY", width=854, height=480, fs=1)
print("Video available at https://youtu.be/" + video.id)
video
```

    Video available at https://youtu.be/YoNbc9M92YY

__Environment:__

- The agent experiences the environment in episodes or trials.
- Episodes terminate by transitioning to the inter-trial-interval (ITI) state and they are initiated from the ITI state as well. We clamp the value of the terminal/ITI states to zero.
- The classical conditioning environment is composed of a sequence of states that the agent deterministically transitions through. Starting at State 0, the agent moves to State 1 in the first step, from State 1 to State 2 in the second, and so on.  These states represent time in the tapped delay line representation
- Within each episode, the agent is presented a CS and US (reward).
- The CS is always presented at 1/4 of the total duration of the trial. The US (reward) is then delivered after the CS. The interval between the CS and US is specified by `reward_time`.
- The agent's goal is to learn to predict expected rewards from each state in the trial.

**General concepts**

* Return $G_{t}$: future cumulative reward, which can be written in arecursive form
\begin{align}
G_{t} &= \sum \limits_{k = 0}^{\infty} \gamma^{k} r_{t+k+1} \\
&= r_{t+1} + \gamma G_{t+1}
\end{align}
where $\gamma$ is discount factor that controls the importance of future rewards, and $\gamma \in [0, 1]$. $\gamma$ may also be interpreted as probability of continuing the trajectory.
* Value funtion $V_{\pi}(s_t=s)$: expecation of the return
\begin{align}
V_{\pi}(s_t=s) &= \mathbb{E} [ G_{t}\; | \; s_t=s, a_{t:\infty}\sim\pi] \\
& = \mathbb{E} [ r_{t+1} + \gamma G_{t+1}\; | \; s_t=s, a_{t:\infty}\sim\pi]
\end{align}
With an assumption of **Markov process**, we thus have:
\begin{align}
V_{\pi}(s_t=s) &= \mathbb{E} [ r_{t+1} + \gamma V_{\pi}(s_{t+1})\; | \; s_t=s, a_{t:\infty}\sim\pi] \\
&= \sum_a \pi(a|s) \sum_{r, s'}p(s', r)(r + V_{\pi}(s_{t+1}=s'))
\end{align}

**Temporal difference (TD) learning**

* With a Markovian assumption, we can use $V(s_{t+1})$ as an imperfect proxy for the true value $G_{t+1}$ (Monte Carlo bootstrapping), and thus obtain the generalised equation to calculate TD-error:
\begin{align}
\delta_{t} = r_{t+1} + \gamma V(s_{t+1}) - V(s_{t})
\end{align}

* Value updated by using the learning rate constant $\alpha$:
\begin{align}
V(s_{t}) \leftarrow V(s_{t}) + \alpha \delta_{t}
\end{align}

  (Reference: https://web.stanford.edu/group/pdplab/pdphandbook/handbookch10.html)

__Definitions:__

* TD-error:
\begin{align}
\delta_{t} = r_{t+1} + \gamma V(s_{t+1}) - V(s_{t})
\end{align}

* Value updates:
\begin{align}
V(s_{t}) \leftarrow V(s_{t}) + \alpha \delta_{t}
\end{align}

## Exercise 1: TD-learning with guaranteed rewards

Implement TD-learning to estimate the state-value function in the classical-conditioning world with guaranteed rewards, with a fixed magnitude, at a fixed delay after the conditioned stimulus, CS. Save TD-errors over learning (i.e., over trials) so we can visualize them afterwards.

In order to simulate the effect of the CS, you should only update $V(s_{t})$ during the delay period after CS. This period is indicated by the boolean variable `is_delay`. This can be implemented by multiplying the expression for updating the value function by `is_delay`.

Use the provided code to estimate the value function.

```python
def td_learner(env, n_trials, gamma=0.98, alpha=0.001):
  """ Temporal Difference learning

  Args:
    env (object): the environment to be learned
    n_trials (int): the number of trials to run
    gamma (float): temporal discount factor
    alpha (float): learning rate

  Returns:
    ndarray, ndarray: the value function and temporal difference error arrays
  """
  V = np.zeros(env.n_steps) # Array to store values over states (time)
  TDE = np.zeros((env.n_steps, n_trials)) # Array to store TD errors

  for n in range(n_trials):
    state = 0 # Initial state
    for t in range(env.n_steps):
      # Get next state and next reward
      next_state, reward = env.get_outcome(state)
      # Is the current state in the delay period (after CS)?
      is_delay = env.state_dict[state][0]

      ########################################################################
      ## TODO for students: implement TD error and value function update
      # Fill out function and remove
      #raise NotImplementedError("Student excercise: implement TD error and value function update")
      #################################################################################
      # Write an expression to compute the TD-error
      TDE[state, n] = (reward + gamma * V[next_state] - V[state])

      # Write an expression to update the value function
      V[state] += alpha * TDE[state, n] * is_delay

      # Update state
      state = next_state

  return V, TDE

# Uncomment once the td_learner function is complete
env = ClassicalConditioning(n_steps=40, reward_magnitude=10, reward_time=10)
V, TDE = td_learner(env, n_trials=20000)
learning_summary_plot(V, TDE)
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W2D5_ReinforcementLearning/solutions/W2D5_Tutorial1_Solution_0d9c50de.py)

*Example output:*

## Interactive Demo 1: US to CS Transfer

During classical conditioning, the subject's behavioral response (e.g., salivating) transfers from the unconditioned stimulus (US; like the smell of tasty food) to the conditioned stimulus (CS; like Pavlov ringing his bell) that predicts it. Reward prediction errors play an important role in this process by adjusting the value of states according to their expected, discounted return.

Use the widget below to examine how reward prediction errors change over time. Before training (orange line), only the reward state has high reward prediction error. As training progresses (blue line, slider), the reward prediction errors shift to the conditioned stimulus, where they end up when the trial is complete (green line).

Dopamine neurons, which are thought to carry reward prediction errors _in vivo_, show exactly the same behavior!

```python
#@title

#@markdown Make sure you execute this cell to enable the widget!

n_trials = 20000

@widgets.interact
def plot_tde_by_trial(trial = widgets.IntSlider(value=5000, min=0, max=n_trials-1 , step=1, description="Trial #")):
  if 'TDE' not in globals():
    print("Complete Exercise 1 to enable this interactive demo!")
  else:

    fig, ax = plt.subplots()
    ax.axhline(0, color='k') # Use this + basefmt=' ' to keep the legend clean.
    ax.stem(TDE[:, 0], linefmt='C1-', markerfmt='C1d', basefmt=' ',
            label="Before Learning (Trial 0)",
            use_line_collection=True)
    ax.stem(TDE[:, -1], linefmt='C2-', markerfmt='C2s', basefmt=' ',
            label="After Learning (Trial $\infty$)",
            use_line_collection=True)
    ax.stem(TDE[:, trial], linefmt='C0-', markerfmt='C0o', basefmt=' ',
            label=f"Trial {trial}",
            use_line_collection=True)

    ax.set_xlabel("State in trial")
    ax.set_ylabel("TD Error")
    ax.set_title("Temporal Difference Error by Trial")
    ax.legend()
```

    interactive(children=(IntSlider(value=5000, description='Trial #', max=19999), Output()), _dom_classes=('widge…

## Interactive Demo 2: Learning Rates and Discount Factors

Our TD-learning agent has two parameters that control how it learns: $\alpha$, the learning rate, and $\gamma$, the discount factor. In Exercise 1, we set these parameters to $\alpha=0.001$ and $\gamma=0.98$ for you. Here, you'll investigate how changing these parameters alters the model that TD-learning learns.

Before enabling the interactive demo below, take a moment to think about the functions of these two parameters. $\alpha$ controls the size of the Value function updates produced by each TD-error. In our simple, deterministic world, will this affect the final model we learn? Is a larger $\alpha$ necessarily better in more complex, realistic environments?

The discount rate $\gamma$ applies an exponentially-decaying weight to returns occuring in the future, rather than the present timestep. How does this affect the model we learn? What happens when $\gamma=0$ or $\gamma \geq 1$?

Use the widget to test your hypotheses.

```python
#@title

#@markdown Make sure you execute this cell to enable the widget!

@widgets.interact
def plot_summary_alpha_gamma(alpha = widgets.FloatSlider(value=0.0001, min=0.001, max=0.1, step=0.0001, description="alpha"),
                             gamma = widgets.FloatSlider(value=0.980, min=0, max=1.1, step=0.010, description="gamma")):
  env = ClassicalConditioning(n_steps=40, reward_magnitude=10, reward_time=10)
  try:
    V_params, TDE_params = td_learner(env, n_trials=20000, gamma=gamma, alpha=alpha)
  except NotImplementedError:
    print("Finish Exercise 1 to enable this interactive demo")

  learning_summary_plot(V_params,TDE_params)

```

    interactive(children=(FloatSlider(value=0.001, description='alpha', max=0.1, min=0.001, step=0.0001), FloatSli…

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W2D5_ReinforcementLearning/solutions/W2D5_Tutorial1_Solution_12ce49be.py)

---
# Section 2: TD-learning with varying reward magnitudes

In the previous exercise, the environment was as simple as possible. On every trial, the CS predicted the same reward, at the same time, with 100% certainty. In the next few exercises, we will make the environment more progressively more complicated and examine the TD-learner's behavior.

## Interactive Demo 3: Match the Value Functions

First, will replace the environment with one that dispenses one of several rewards, chosen at random. Shown below is the final value function $V$ for a TD learner that was trained in an enviroment where the CS predicted a reward of 6 or 14 units; both rewards were equally likely).

Can you find another pair of rewards that cause the agent to learn the same value function? Assume each reward will be dispensed 50% of the time.

Hints:
* Carefully consider the definition of the value function $V$. This can be solved analytically.
* There is no need to change $\alpha$ or $\gamma$.
* Due to the randomness, there may be a small amount of variation.

The average is 10

```python
#@title

#@markdown Make sure you execute this cell to enable the widget!

n_trials = 20000
np.random.seed(2020)
rng_state = np.random.get_state()
env = MultiRewardCC(40, [6, 14], reward_time=10)
V_multi, TDE_multi = td_learner(env, n_trials, gamma=0.98, alpha=0.001)

@widgets.interact
def reward_guesser_interaction(r1 = widgets.IntText(value=0, min=0, max=50, description="Reward 1"),
                               r2 = widgets.IntText(value=0, min=0, max=50, description="Reward 2")):
  try:
    env2 = MultiRewardCC(40, [r1, r2], reward_time=10)
    V_guess, _ = td_learner(env2, n_trials, gamma=0.98, alpha=0.001)
    fig, ax = plt.subplots()
    m, l, _ = ax.stem(V_multi, linefmt='y-', markerfmt='yo', basefmt=' ', label="Target",
            use_line_collection=True)
    m.set_markersize(15)
    m.set_markerfacecolor('none')
    l.set_linewidth(4)
    m, _, _ = ax.stem(V_guess, linefmt='r', markerfmt='rx', basefmt=' ', label="Guess",
                      use_line_collection=True)
    m.set_markersize(15)

    ax.set_xlabel("State")
    ax.set_ylabel("Value")
    ax.set_title("Guess V(s)\n" + reward_guesser_title_hint(r1, r2))
    ax.legend()
  except NotImplementedError:
    print("Please finish Exercise 1 first!")
```

    interactive(children=(IntText(value=0, description='Reward 1'), IntText(value=0, description='Reward 2'), Outp…

## Section 2.1 Examining the TD Error

Run the cell below to plot the TD errors from our multi-reward environment. A new feature appears in this plot? What is it? Why does it happen?

```python
plot_tde_trace(TDE_multi)
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W2D5_ReinforcementLearning/solutions/W2D5_Tutorial1_Solution_dea47c05.py)

---
# Section 3: TD-learning with probabilistic rewards

In this environment, we'll return to delivering a single reward of ten units. However, it will be delivered intermittently: on 20 percent of trials, the CS will be shown but the agent will not receive the usual reward; the remaining 80% will proceed as usual.

 Run the cell below to simulate. How does this compare with the previous experiment?

Earlier in the notebook, we saw that changing $\alpha$ had little effect on learning in a deterministic environment. What happens if you set it to an large value, like 1, in this noisier scenario? Does it seem like it will _ever_ converge?

```python
np.random.set_state(rng_state) # Resynchronize everyone's notebooks
n_trials = 20000
try:
  env = ProbabilisticCC(n_steps=40, reward_magnitude=10, reward_time=10,
                        p_reward=0.8)
  V_stochastic, TDE_stochastic = td_learner(env, n_trials*2, alpha=.001)
  learning_summary_plot(V_stochastic, TDE_stochastic)
except NotImplementedError:
  print("Please finish Exercise 1 first")
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W2D5_ReinforcementLearning/solutions/W2D5_Tutorial1_Solution_cbbb9c00.py)

---
# Summary

In this notebook, we have developed a simple TD Learner and examined how its state representations and reward prediction errors evolve during training. By manipualting its environment and parameters ($\alpha$, $\gamma$), you developed an intuition for how it behaves.

This simple model closely resembles the behavior of subjects undergoing classical conditioning tasks and the dopamine neurons that may underlie that behavior. You may have implemented TD-reset or used the model to recreate a common experimental error. The update rule used here has been extensively studied for [more than 70 years](https://www.pnas.org/content/108/Supplement_3/15647) as a possible explanation for artificial and biological learning.

However, you may have noticed that something is missing from this notebook. We carefully calculated the value of each state, but did not use it to actually do anything. Using values to plan _**Actions**_ is coming up next!

# Bonus

## Exercise 2: Removing the CS

In Exercise 1, you (should have) included a term that depends on the conditioned stimulus. Remove it and see what happens. Do you understand why?
This phenomena often fools people attempting to train animals--beware!

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W2D5_ReinforcementLearning/solutions/W2D5_Tutorial1_Solution_a35b23f3.py)