Good move getting a nice robot! I'm doing something similar to you, but I went with a cheap robot, the "HIWONDER 6DOF Robotic Arm Kit". It was only $600, but wow is it bad. The precision and repeatability are both "are you drunk?" level. I can hear the gears grind when it moves. I suppose I should upgrade. But if my system can work with a terrible robot, I assume it would work even better with a nice one!
That video is pretty good, thanks for finding it. I'm basically betting that an earlier, abandoned approach described in the video, "Code as Policy", will beat everything else. It requires no training data, and generalizes instantly to all robots.
I was also just in the market for a small experiment robot. I got the hiwonder armpi-fpv. Avoid it, the actuators are pretty bad - they're very 'grindy', the robot jitters like crazy when it moves. Any such problems with the lekivi?
For the fine-motor commands: or, the model can write the code to generate them on the fly. It seems to work, in my very limited experiment.
As for memory: my approach is to give the robot a python repl and, basically, a file system - the LLM can write modules, poke at the robot via interactive python, etc.
Basically, the LLM becomes a robot programmer, writing code in real-time.
This seems perfect to hook up to my 'LLMs can control robots over MCP' system. The idea is that LLMs are great at writing code, so let's lean in to that. I'll give it a try! I just got a bigger robot, we'll see how it does...
Remember how all those years ago (oh wait, a few months ago), that big npm hack attack was found by someone thinking their ssh connection was half a second too slow? Hope this ain't like that!
Yeah, the mechanism by which the LLMs control the robot is by writing code. I suppose they could also issue direct joint sequences, but I thought that they're so good at writing code already, might as well do that. So if they 'wanted' to they could write code with an explicit joint sequence they calculate in-context. That one seems more difficult.
So they can go 'slow', by taking a camera image, controlling the robot, repeating. Or they can write code that runs closer to the robot in a loop, either way. I thought the latter was somehow more impressive, and that's what you see in the hand-tracking example.
I'm connecting LLMs directly to robots, to see how well they can perform robot things by directly controlling motors and sampling the camera/sensors. Initial results are encouraging!
Here's an idea for how to do that: treat frontier AI as a sort of 'common carrier'. The only business that frontier AI labs are allowed to conduct is selling raw tokens - no UI. Thus, 'claude code' would have to come from some other company. This would segment the AI industry, and, maybe, prevent a single entity (or small number of entities) from capturing all value.
Yet another problem with MCP: every LLM harness that does support it at all supports it poorly and with bugs.
The MCP spec allows MCP servers to send back images to clients (base64-encoded, some json schema). However:
1) codex truncates MCP responses, so it will never receive images at all. This bug has been in existence forever.
2) Claude Code CLI will not pass those resulting images through its multi-modal visual understanding. Indeed, it will create an entirely false hallucination if asked to describe said images.
3) No LLM harness can deal with you bouncing your local MCP server. All require you to restart the harness. None allow reconnection to the MCP server.
I assure you there are many other similar bugs, whose presence makes me think that the LLM companies really don't like MCP, and are bugly-deprecating it.
I've got one of these! Mine is called 'roboflex' (github.com/flexrobotics). It's c++/python, not rust. But similarly born out of frustration with ros. Writing your own robotics middleware seems to be a rite of passage. Just like 'writing your own game engine'. Nothing wrong with that - ros is powerful but has legit problems, and we need alternatives.
Although tbh, these days I'm questioning the utility. If I'm the one writing the robot code, then I care a lot about the ergonomics of the libraries or frameworks. But if LLMs are writing it, do I really care? That's a genuine, not rhetorical question. I suppose ergonomics still matter (and maybe matter even more) if I'm the one that has to check all the LLM code....
My project is https://github.com/colinator/Ariel - basically, no VLAs - instead, "just write code". Or have the agents do it.
I don't have a writeup yet about applying Ariel to _this_ robot, but this is for a previous one: https://colinator.github.io/Ariel/post1.html.
Excited to follow your progress!