It will work for downloaded videos as soon as someone develops a video player app that supports side by side 3D or vertical 3D, along with a 180°-360° field of view (FOV), to allow one look around while watching the video. The way 3D VR content with a wide field of view works is that the video player application takes a video file designed with this in mind (you can see that if you play the video file in a normal media player like VLC, it will look all stretched out and mirrored horizontally or vertically) and renders it on a set canvas with the correct FOV and 3D implementation selected. On Android there are plenty of apps that do this for Google cardboard/daydream and the Samsung VR headset.

As for watching videos on the internet and streaming them, that's more complex. It usually involves turning on some browser flags and often does not work correctly. This is for any kind of 3D or wide FOV video, so downloading first and watching using an appropriate video player is the best way to get a consistent experience. I remember having to do this for rollercoaster videos back when Google cardboard was new.