The creator of that repo has a blog with some posts on building a wasm version of FFmpeg, I mostly followed those: https://jeromewus.medium.com/
I also used some of their scripts for building the 3rd-party libraries.
It took a bit of trial and error to see which versions of the different libraries and build tools work together.
You don't need to keep the video files in memory, it's enough to have the javascript File object and then you can read the packets you need on the fly. This way the app's memory usage should be okay.
This is what I did in my similar project (vidmix.app) and seems to be what OP's project is doing, except that they also copy the file to indexedDB to keep it around after the browser tab is closed.
It seems to me that the demonstration is calculated in sRGB space, so with non-linear brightness values and I suspect that most of the unnaturalness of the smear is due to that. To simulate the physics this would need to be done with linear brightness values and only at the end converted to sRGB.
(unless some non-linear effects in human visual perception cancel all that out, but it should at least be mentioned if so)