michel-pohl/rtrl_time_series_forecasting

Online time-series forecasting and 3D image reconstruction using RTRL-trained vanilla RNNs and Lucas–Kanade optical flow

This page documents the reference implementation used in the following paper on online training of vanilla RNNs using Real-Time Recurrent Learning (RTRL) for medical time-series forecasting and 3D image recontruction.

Paper

• Title: Prediction of the motion of chest internal points using a recurrent neural network trained with real-time recurrent learning for latency compensation in lung cancer radiotherapy
• Authors: Michel Pohl, Mitsuru Uesaka, Kazuyuki Demachi, Ritu Bhusal Chhatkuli
• arXiv: https://doi.org/10.48550/arXiv.2207.05951
• Published version: https://doi.org/10.1016/j.compmedimag.2021.101941
• Hugging Face paper page: https://huggingface.co/papers/2207.05951

Code

This paper relies on several components, implemented in separate repositories:

• Time-series forecasting with RTRL (core ML module)
  • GitHub: https://github.com/pohl-michel/Time-series-prediction-with-an-RNN-trained-with-RTRL
  • Latest release (Zenodo): https://doi.org/10.5281/zenodo.4452209
• Optical flow estimation (Lucas-Kanade optical flow) for 3D image sequences
  • GitHub: https://github.com/pohl-michel/Lucas-Kanade-pyramidal-optical-flow-for-3D-image-sequences
  • Latest release (Zenodo): https://doi.org/10.5281/zenodo.4011768
• Image warping via Nadaraya–Watson regression
  • GitHub: https://github.com/pohl-michel/3D-image-warping-using-Nadaraya-Watson-non-linear-regression
  • Latest release (Zenodo): https://doi.org/10.5281/zenodo.4011749

Description

We provide an implementation of the pyramidal iterative Lucas-Kanade optical flow algorithm for 3D image sequences, which we use to estimate motion in artificially extended 4D chest imaging acquisitions. We forecast the position of three internal points near the lung tumor using RTRL, and use a simple linear correspondence model to recover the future dense deformation field within the selected ROI. Nadaraya-Watson non-linear regression is used to warp the initial 3D reference frame according to the predicted motion field. Although grid search on the validation set was resource intensive, inference time remained acceptable, and the tumor position in the predicted images was qualitatively consistent with the ground truth. Overall, this work addresses respiratory motion forecasting for treatment-system latency compensation in radiotherapy, with a focus on online learning for continuous adaptation to unsteady breathing dynamic during radiation delivery.

Citation

@article{pohl2021prediction,
  title={Prediction of the motion of chest internal points using a recurrent neural network trained with real-time recurrent learning for latency compensation in lung cancer radiotherapy},
  author={Pohl, Michel and Uesaka, Mitsuru and Demachi, Kazuyuki and Chhatkuli, Ritu Bhusal},
  journal={Computerized Medical Imaging and Graphics},
  volume={91},
  pages={101941},
  year={2021},
  publisher={Elsevier}
}