I agree with this. I recently went back to writing more code by hand again for this reason: the amount of effort / mental toll it takes me to steer the LLM is sometime just not worth it. I might be slower but I enjoy my work much more again.
- There is no double taxation if you just pay yourself a salary (since it’s a normal business expense). If you want to take money out of the company flexibly, a GmbH is the wrong structure.
- I’ve never heard of anybody doing an UG/GmbH + KG to get started. This is highly unusual. Most people either do just a simple UG or maybe they set up a holding structure with two separate GmbH / UG entities.
- Related to the above: if you go with a simple, standard structure you will incur minimal legal fees. You don’t need a lawyer, you just directly task a notary and tell them you want a standard setup.
- If you don’t want the complexity of a limited liability company, the standard way to reduce liability risk is to get liability insurance. Many, many people do this instead of having a GmbH.
The valid criticism is the a) lack of digital processes and b) sequential processing of steps that could happen in parallel. For example, I sped up my own GmbH process by driving to the register court and paying in cash on-site. For whatever reason that’s much faster and saves about a week.
Yeah I tried to use ROS / Gazebo about 10 years ago (it was still ROS 1) and getting it set up was an immense pain. I remember that creating new modules required writing CMake files. Maybe this is better now, but I decided to skip that.
I also agree on the need for a simple, easy, extensible open source framework. LeRobot IMO is some of this but also contains the dataset + ML code. I think Flask is nice because it's so singularly focused on just one thing with extensibility if you need extras.
So I really like the idea. But being an OSS maintainer these days seems... intense.
For depth I agree on the VLA route but for ACT / DP-style imitation learning from scratch it seems more feasible (since you’re not fighting a pretrained model that was not trained on this modality). Might also increase robustness since you naturally end up with an input that’s invariant to colors / textures. Plan is to try both paths: the from scratch (and then ablate RGB vs RGB-D) and the VLA + fine-tuning one.
Did you ever a late RGB vs RGB-D? I haven’t seen that much in the literature.
Also any thoughts on action space representation? Seems to me people are settling on flow matching mostly, but pi still uses discrete tokens to supervise the upstream backbone VLM. I also like the simplicity of discrete bins and used that successfully in the past.
Yeah I can see that. Especially the rotation part is pretty awkward (hence why I currently do the axis parallel to table surface trick and only use the yaw angle to rotate the gripper). The space mouse was super easy to get started with though, so in terms of getting something up and running quickly it has an advantage.
This seems indeed useful. I haven't seen this for robotics but I'm sure people need this for larger deployments (either for a distributed fleet or for an "arm farm"-like setup where there's many robots in one location for data collection / eval). Interesting idea!
Oh very cool! Looks a bit like the TRLC-DK1 (I was looking at this one for a bit).
I think pushing the sota is quite hard to do solo but we'll see. Mostly I want to get back up to speed after having not done much robotics during the last 6 years. Best way for me to learn is to just do it, so here we are. We'll see how far I get (I suspect at some point compute will be the main bottleneck)
I timestamp everything twice: once with the hardware clock (if available, like for the realsense camera) and once within my robot stack once it gets read from the device (using `time.monotonic_ns()`). Both are stored and alignment can happen with either timestamp. I think the 2nd timestamp is actually more meaningful since ultimately I want to reconstruct the state that the policy would've seen; so if one modality is delayed I should actually include that effect during training.
That being said, I might switch to a realsense for the static tabletop camera as well; the realsense wrist is clearly much more reliable than the cheap Logitech C920 that I currently use.
I'm only starting down this road but my sense is that ACT and Diffusion Policy both make it pretty feasible to start on real data only. LeRobot also makes it easier to train these. But that's the next step that I'm working on, so I don't know yet.
On attachments: during this project I really wanted a 3D printer several times. So that's probably next on the shopping list.
The Ufactory arm is actually quite extensible: it exposes digital input/output and you have a standard wrist mount where you can mount different end effectors or attachments.
And yeah I feel you re humanoid. I worked on the Rubik's cube project at OpenAI, which used a humanoid hand, and it was insanely painful and hard. Also fun anecdote: it was completely impossible to teleop the shadow hand. We had a data glove to capture hand movements but as soon as contact / haptics come in, you're lost. We could never even get a single rotation on the Rubik's cube via teleop.
I do think simpler hardware like the one described in my post works really though and it's so much easier to do something with it.
Great points and I very much appreciate the input!
Right now the static camera is probably really bad: It's mounted on my desk, so its very easy to bump into it and move it. So yeah, it's position for sure will change over time. I think I need a better solution, maybe a rail system that's more rigidly attached to the robot arm so that at least the camera stays fixed relative to that point of reference.
For understanding: I think the level is much deeper if I wrote the code vs reading someone else’s. Same applies to coding agents of course which is why I wrote most of it myself and only delegate some tasks (for example codex was great help at setting up telemetry dashboards or writing the custom glfw renderer).
On control: LeRobot will change all the time and I’ll be unaware of what changed. If something suddenly doesn’t work anymore, it’s a pain to find out. I can of course fork or pin but that defeats the purpose a bit.
At the end it’s also partially just preference: I wanted to write this layer myself, and I have opinions about how it should be architected, so I did.
- The driver situation turned out totally fine; I intentionally picked HW with good python sdk support so that was very painless.
- The static camera (the C920) is not super great; it drops frames and sometimes cuts out. We’ll see how that goes but it’s probably the clostest thing I want to swap right now. Another issue is reach of the arm when forcing the worst to be axis parallel with the table; you cannot get very far away. The chess setup demo in the video gives an example: I can just reach the row of pawns and beyond that it’s out of reach.
- I don’t know yet! The 50-100 figure comes from the ACT and diffusion policy papers but it depends on the type of task. For fine tuning my sense is that you only need a few hours worth of demos to get good results with pi0.5 etc. a big reason I’m doing this project is that I want to try all of this myself, so the next posts definitely will talk about that
Looks very cool! I’m not a huge discord user but how about you shoot me an email and we can figure out how to share notes? (I don’t want to post it directly here but it’s easy to find on my personal website, just google my name)
Re why not SO-101: the article has a footnote about this; I actually bought the SO-101 as well! I want to integrate it into the same setup so I can switch depending on task.
Somewhat surprisingly the xarm was actually much faster to arrive; I got it within 2 days of ordering. I don’t have a 3D printer and getting the SO-101 from the vendor I ordered it at took almost 4 weeks. So partially it just came down to what I had access to more quickly.
Second point is reliability: I think the SO-101 is cool but I’d be surprised if it doesn’t break more quickly than the xarm. I wanted something that’s going to last a long time without headaches. And these industrial arms are really mature hardware wise now.