* We made an open source C++ NeRF volumetric video engine for VR.
* It trains a NeRF for each frame of video, then compresses it to our format 'ldi3', which enables real-time playback on Quest or Vision Pro in WebXR, Unity or Unreal (open source players are available for all of these).
* We provide a paper explaining how to bake NeRFs into layered depth images for real time rendering.
* This is a commercial product. It is free to try, but requires a license to render the final results as a video output.
* It can output 2D videos where a virtual camera flys along a smooth path, or VR180 and Looking Glass portrait holograms.
* If I'm not mistaken this is the first NeRF-based VR stitching engine.
* Unlike NeRF Studio, Volurama does not require command line tools to install.
* Unlike Luma.ai, Volurama is a Windows/Mac application, which means you process your data locally rather than uploading it to the cloud.
* This is a custom/from scratch NeRF and structure-from-motion engine written in C++. It doesn't depend on COLMAP or NVidia's tinycudann. It incorporates ideas from several recent publications, as well as a few proprietary tricks developed at Lifecast.
* This is a 1.0 alpha release, and there are bound to be some bugs.
Today Lifecast unveils text-to-full 3D immersive environments that can be viewed in VR (e.g., Quest 2) or on 2D screens. We are doing this with a combination of Stable Diffusion and several other neural nets to make it 3D, combined with Lifecast's format for 6DOF VR photos and video. It's free to try and we do the processing in the cloud. Check it out and tell us what you think! This is version 1.0 and we are iterating quickly, so expect improvements in the future.
Artifacts at the edges are due to occlusions. An occlusion is a part of the scene which wasn't visible to the original camera. You see these if you move far from the original camera in VR to look behind something. This is a really hard problem for 6DOF. We've been improving the quality of occlusions over time, e.g.:
* v1: https://lifecastvr.com/demo_maui.html
* v2: https://lifecastvr.com/kalalea_fire.html
* v3: https://lifecastvr.com/hubner4.html
Version 3 now uses a 2-layer representation which has an image+depthmap for the background layer, which is drawn to fill in the occlusions. This background layer can be precomputed in a variety of different ways. For example, here is a CGI synthetic scene where we can construct the background layer perfectly:
https://lifecastvr.com/liferay.html
However, making up the background layer for real-world content is more challenging. We are on version 1 of that. We will improve this with machine learning in a future release. We can also substitute a "plate" 3d scene for the background in cases where the camera doesn't move. We have also experimented with using data from other frames when the camera moves. This will improve over time.
I have seen a few similar approaches. Basically, have several views with corresponding depth map, then render all of them and blend. As with MPI, the problem is the Quest 2 doesn't have enough GPU power to do the blending (in my tests at least). So I will be pleasantly surprised if this method runs on Quest 2, and even more surprised if it shows up on Quest 2 + WebVR. There are also limits of video decode resolution, so more views means lower pixel density in the output; VR video professionals are extremely sensitive to this. So an approach like this works better for photos than video.
In contrast, we do one view with 2 layers, and we composite them with no blending. This rendering/encoding is optimized to work with the limits of Quest 2.
I worked on a lightfield(ish) camera at Facebook, but Lifecast is more focussed on what is practical with current camera hardware. We prefer to make the best possible 6DOF using existing VR180 cameras which people already have. A second challenge is to render the results on a Quest 2 (the most popular VR headset today), with its limited GPU power. Our format is optimized for the rendering capabilities of Quest 2, which means we have to make some tradeoffs on visual quality. I don't think Quest 2 has enough power to render multi-layer images (MPIs, the format in DeepView). This is the difference between making a product and doing academic research. I'm looking forward to Quest 3 or whatever comes next; I hope it has enough power to do stuff like MPI.