ScrollyVideo.js

You’ve probably seen it in a fancy visual journalism piece from a well-known media organization: the scrolling video.

Jan 2023

You’ve probably seen it in a fancy visual journalism piece from a well-known media organization: the scrolling video. On first thought, it seems simple enough: use Javascript to figure out how far down the page the user has scrolled, and programmatically move the playhead of the video to the appropriate point so that the video reacts to the user’s scroll event.

But it’s not that easy (discussed below), and I’m finally releasing an open source library with components in React, Svelte, Vue, and vanilla javascript to make scrolling video projects easier. Early versions implementations can be seen in projects I did for Commonwealth Magazine, or even in this ProPublica report. This was a project sitting under wraps for nearly year, as it got lost under much of my other day-to-day responsibilities. I finally found the time to clean up the documentation, test different frameworks, and get this project out the door.

CurrentTime

Unfortunately, video formats were never designed with this use case in mind, and will often take seconds or longer to load the frame under normal circumstances. Used in a scrolling video, this results in a horribly choppy experience. The reason this happens is videos are typically encoded using keyframes set every 30 frames or so. In videos, keyframes are frames that contain the pixel data for the entire frame, whereas frames that are not keyframes only hold the “difference” between this frame and the last frame. Having frames only encode frames that have changed allows the video to be compressed to a smaller size, as most videos are only expected to be played forwards.

Keyframes allow video frames to only store the difference from frame to frame.

Visualizing a video frame simply storing the difference between frames.

Therefore, when it comes to exporting video for a “scrolly” use case, the recommendation is to export it with the setting keyframes=1, which tells the encoder that every single frame is a keyframe. While this solves our problem of allowing the video to dynamically load the right frame of the video much faster, it also causes the size of the video file to increase significantly, all other settings being equal. In my experience, going down this path will result in re-exporting the video multiple times while adjusting the quality setting to find a compromise between file size and video quality, which is not ideal.

PlaybackRate

After experimenting with this method for a while, I found a second approach: simply playing and pausing the video while dynamically adjusting the playback rate. If you’ve ever played around with video player settings, you’ll know that videos on the web often have the option of changing the playback speed of the video, allowing you to power through a lecture at two or three times speed. In fact, most web browsers support up to eight times speed, a speed that I have no honest idea when one would actually want to use in real life. Using playback rate, I can essentially mimic the effect of a user scrolling fast or slow, while relying on the video player to decode the frames in order, making the forward scrolling experience extremely smooth.

However, the catch with this method is that playback rate cannot be a negative number, so scrolling backwards must still be done with the first method above. Theoretically you could export an identical video in reverse and have two video elements that show or hide depending on the scroll direction, but scrollyVideo.js currently does not support this option. Additionally, Safari for some reason is less performant using this approach than the one above, so this library detects Safari and forces it to use the first method.

WebCodecs

The final approach I stumbled across was using the WebCodecs API to convert a video into individual frames in the browser. Unfortunately, WebCodecs is only supported in Chrome at the moment, with no estimated release in any of the other browsers. And while I did find a polyfill for WebCodecs, I was unable to get it working with ScrollyVideo, so this method is limited to Chromium-based browsers only.

Essentially, by reading all the frames from a video ahead of time, this method is able to have any possible frame immediately ready for painting to a canvas. The downside? It takes a bit of time before the video is fully processed, so any immediate usage of this method will likely fall back to one of the earlier ones. Going this route also requires more memory and processing power, something that lower-end android devices may not handle well.

Additional Use Cases

For more creative use cases, scrolling may not be the only way that a project may want to control the playback and position of a video. Perhaps you want to control the position of the video based on mouse movement or something else. By exposing setCurrentTimePercent from the library, you can also directly set the position the video.

Frameworks

That said, the implementation of this project was built with a vanilla javascript installation in mind, with all the logic living inside ScrollyVideo.js. The React, Svelte, and Vue components are simply wrappers around the plain javascript implementation, which turns out to be much easier than trying to create a WebComponent.