Oh yes. This could be a nice interface to provide other inputs we (plan2scene) want as well. Like photo to room assignments. Then, plan2scene can create 3d houses with appropriate materials. Thank you for sharing this link.
Do you think its possible to decouple lighting effects from the generated 3D scene? Currently it seem to simply copy the actual lighting condition in the environment. In a way, it's a limitation. It's much better if it can provide the RGB color + other material information (SVBRDF decomposition). If that's possible, then these meshes can be used in other environments.
This can detect rooms as polygons, doors/windows as line segments and bounding boxes around objects (e.g. cabinets, sinks etc. indicated on floorplans).
For instance, look at this sample 3d scanned house: https://matterport.com/en-gb/media/2486. Make sure to switch to the doll-house view from the toolbar. There are many holes in the scan.
This is definitely a thought I have considered. It could potentially ease lot of hassle related to 3D scanning of houses by real-estate companies. Very often, those scans require expensive equipment, professional expertise to use those equipment and considerable time and effort to capture the entire residence. Even then,you will find many holes in those scans and floating debris. With our work, one only needs a sketch of the floorplan and photos taken from that house. Plan2Scene will use data to infer appearance of unseen surfaces.
Estimating the global camera pose using non-overlapping photos is a challenging problem. This is doable if the photos densely cover the house (e.g. video frames).
However, if the global camera poses are available, we can detect surfaces in photos and project them reasonably well into 3D. In the past, I have used this work to achieve something similar: https://github.com/NVlabs/planercnn