These are really really interesting thoughts around the remote teleop interface. A few things you mentioned have been on my mind. Teleop UX is underdeveloped and its affects is underestimated, and it could turn out to be a huge thing for us and humanoid companies if autonomy is harder than it seems now.
I don't believe construction equipment operation today is optimal. We just go with it as that's what we have. It's far from intuitive and easy for a newbie to start operating to give some context and requires hours of practice. There would be a bit of training to be done for remote teleop as well. Might as well make it easy to interface and improve on the existing experience, which hasn't really evolved for years and decades!
I'd like to have a chat with you if you're up for it: contact at useflywheel dot ai
I completely agree. If you have worked in the industry, I would like to talk to you and get your feedback on what I might be missing on my understanding in requirements in safety layers.
Would be great if we can chat on email: contact at useflywheel dot ai
This is a great idea - I would dig into this.
I disagree on AI not winning here. Support crew mostly work on eliminating blindspots for operator. Yes you can eliminate blindspots with cameras and a 7inch TFT screen, but those are small, don't work great in direct sunlight. The current solution not being 10x better is why they still fallback to having those crew members. I genuinely believe we will change that.
I totally agree, what I meant was if we only do CANbus retrofit, we would really limit ourselves in how compatible we are. So we have to focus on doing the right kind of retrofit really well. CANbus retrofit is easier and we've done that many times before. Especially because my cofounder Mahimana used to work in self-driving company and worked on this and single-handedly built out their current ML data and eval pipeline.
You've touched one of my favorite things to think about. I think about this a lot. There will always be people who enjoy fully analog cars/planes/watches over alternatives. I think that doesn't define the whole market. How many of new joinees are we seeing in this industry? Would you rather have a functioning back because you operate these remotely or go to site and sit in one of these and have a broken back by 50s?
This is leaving out the fact that remote operation could save lots of lives.
In our launch video we did put together some of the news clips on it: https://www.ycombinator.com/launches/O8i-flywheel-ai-waymo-f...
I agree - the dumping and digging itself (where you move boom, arm, bucket much more than swing/tracks) should be manual. But swinging to the truck and back to pit (pure swinging motion to center around these areas of interest) do not have to be manual. I agree with your and other comments that the transition has to be smooth and that's something we're working on.
I think swinging (which is about 40% of dig and dump workflow by time spent) should not be manual. That's lowest levels of autonomy which requires roughly centering to the pit/truck which we have already achieved. Hence operator only has to look in front!
I'd like to add that this was done in partnership with Frodobots.ai (https://frodobots.ai) and BitRobot.ai (https://bitrobot.ai). They are working on a great mission statement and were amazing partners!
I definitely agree with you - recreating the scene in teleop is challenging. In excavators, however, it does make it better. An excavator has huge blindspots on the right (due to arm), to the back and sometime near the bucket. Hence the workers who are hired to stand around (banksmen, spotters, signalmen) and signal to the operator.
It's like driving a Ford F150 without backup camera. You'd add the backup camera upfront, and not display the back view at the back window.
It's definitely challenging and we're far from something that's perfect. We're iterating towards something that's better everyday.
I can relate to how you felt. Even though I developed this myself, when I saw it running remotely with no one sitting inside for the first time, it felt magical and weird. This is going to save lives and look magical and that's too much fun to put off and do something else!
The way I think about this - we should not have multi screens. Human field of vision is 60 degrees for central and about 120 degrees binocular. The bucket of the excavator is way narrower than this which means actual task doesn't require wide vision.
So if we are able to have really good autonomous safety layers to ensure safe movements, and dynamically resize remote teleop windows, you'd make the operator more efficient. So while we stream 360 degree view, we get creative in how we show it.
That's on the vision side. We also stream engine audio, and do haptic feedback.
Takeuchi are interesting! Rare ones to have blades even on bigger sizes - is that why you got one?
Mining has had autonomous haulage (and to some extent excavation) for some years. They have been very tightly coupled to the individual machines, models and the quarry site.
Construction excavation workflows tend to be different and more varied (hence you would see more variations in the machines/attachments available for this industry). And so far there's no solution which just works or is plug and play. This is an attempt to solve that.
And we believe this is great time to do so due to market demands too. No way Gen Z is going to do these jobs when you can make money being delivery partner/influencer, etc.
The high-certified, high-skilled workforce is going to shockingly shrink in some years.
I might be less experienced than you - I've operated upto 38 tons on maximum 15 degrees incline. I wasn't moving tracks a lot when I did that. Would like to hear what scenarios you'be been in and how would you describe your experience? Maybe I could try those out to learn more!
I think force feedback is key for small excavators, but not really true for 25+ tons excavators. Hence how easy it is for operators to accidentally kill someone with it.
That's indeed what we're trying to test to the extreme - to see how far we could go with just vision. We haven't done heavy excavation workflows yet, but we have some early success with some excavation workflows with just vision input and joystick action output (even without joint angle feedback!).
We're betting on having really huge data with compact observation input and experiment to see if it holds water. If not we can always dial it down and add more sensors/feedback.
Thanks! Yes we're testing both CAN spoofing and retrofits. CAN spoofing really limits compatibility to a few machines/brands. Would love to get in touch with you: my email is contact ai useflywheel dot ai
We're indeed streaming the audio and have haptic feedback. My hypothesis is the seat vibration isn't as helpful. It's sub optimal and operators would be far more productive without it. We would do a paper when we have enough data on this.
We're also putting more cameras but streaming a lot of them at once is tricky.
I'd like to have a chat with you if you're up for it: contact at useflywheel dot ai