If you capture a video and SLAM map of the whole space, you could use some VQA model like cosmos reason offline to extract key points and descriptions. Maybe even plan the route offline for the open ended task like “clean kitchen”. Then load the route and all you need is localization and obstacle avoidance
Admittedly, my use of CUDA and Metal is fairly surface-level. But I have had great success using LLMs to convert whole gaussian splatting CUDA codebases to Metal. It's not ideal for maintainability and not 1:1, but if CUDA was a moat for NVIDIA, I believe LLMs have dealt a blow to it.
To do gaussian splatting anywhere near in real time, you need good depth data to initialize the gaussian positions. This can of course come from monocular depth but then you are back to monocular depth vs lidar.