This page documents the reference implementation used in the following paper on chest and liver cine-MRI frame forecasting with transformers and online training of vanilla RNNs.

This work compares linear filters, sequence-specific and population transformer encoders and online learning algorithms for RNNs (real-time recurrent learning [RTRL], unbiased online recurrent optimization, decoupled neural interfaces, and sparse one-step approximation [SnAp-1]) to forecast thoraco-abdominal cine-MR image sequences. PCA decomposes the motion in the input video, estimated using the Lucas-Kanade optical-flow algorithm, into principal components and time-dependent weights. The latter are predicted using the algorithms above to estimate future motion, and in turn, the future frame, by warping the initial image. RTRL and SnAp- 1 outperformed the other algorithms at medium-to-long horizons. Sequence-specific transformers were competitive for low-to-medium horizons, but transformers remained overall constrained by data scarcity and domain shift between datasets. Predicted frames visually resembled the ground truth, with notable errors occurring near the diaphragm at end-inspiration and regions affected by out-of-plane motion.