| # RNNs / LSTMs for Sequences | |
| ## Resume | |
| Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) networks are sequential neural architectures designed to map time-series data or sequential text structures. LSTMs introduce explicit gating mechanisms to regulate internal information persistence, successfully resolving the vanishing gradient flaws of traditional RNN blocks. | |
| ### Common Use Cases: | |
| * **SMILES Generation:** Autoregressively printing valid chemical strings token by token. | |
| * **Biosignal Analysis:** Evaluating chronological streaming signals from patient telemetry or ECG readouts. | |
| * **Clinical Notes Sequence Modeling:** Tracking medical event timelines over long EHR spans. | |
| ## Content | |
| ### 1. Core Architecture & Gated Memory | |
| Standard RNNs maintain a recurring hidden state vector $h_t$ that gets updated with every timestamp token input. However, backpropagating through long sequences causes gradients to rapidly disappear or explode. | |
| LSTMs solve this by introducing an internal **Cell State** ($c_t$) alongside three specialized gating mechanisms: | |
| * **Forget Gate ($f_t$):** Controls how much historical context from the cell state should be discarded. | |
| * **Input Gate ($i_t$):** Regulates what new contextual state information should be infused into the current cell vector. | |
| * **Output Gate ($o_t$):** Determines what subset of internal hidden cell states should be exposed as the final output block. | |
| ### 2. Operational Execution Sequence | |
| As a sequence executes, the architecture reads the token input at step $t$, merges it with the prior step's hidden vector $h_{t-1}$, adjusts cell memories through the gating parameters, and pushes forward the updated state vectors. This linear memory highway allows the architecture to carry context across extended sequence matrices. | |
| ### 3. Advantages and Disadvantages | |
| * **Advantages:** | |
| * Natural alignment with arbitrary, variable-length chronological or textual stream structures. | |
| * O(L) computational complexity scaling linearly with sequence length. | |
| * **Disadvantages:** | |
| * Strict step-by-step dependency makes parallel sequence acceleration impossible during training. | |
| * Susceptible to information decay over extremely long sequences compared to attention structures. | |
| ### References | |
| * Hochreiter, S., & Schmidhuber, J. (1997). *Long short-term memory*. Neural Computation. | |
| * Segler, M. H., et al. (2018). *Generating focused molecule libraries for drug discovery with recurrent neural networks*. ACS Central Science. |