"Fisheye" generation seems obvious. Give the network a distorted view of an arbitrarily large image, where distant stuff scrunches inward toward a full-resolution point of focus. Predict only a small area - or even a single pixel. This would massively decrease the necessary network size, allowing faster training. (Or more likely, deeper networks). It'd also Hamburger Helper any size dataset by training on arbitrarily many spots within each image instead of swallowing the whole elephant.

Even without that, video only needs a few frames at a time. You want to predict a future frame from several past frames. You want to tween a frame in the middle of past and future frames. That's... pretty much it. Time-lapse "past frames" by sampling one per second, and you can predict the next second instead of the next frame. Then the stuff between can be tweened.