Hey! Your Google Benchmark post was one of my go-to resources when I started picking that up a couple years ago. I love to see the focus on performance benchmarking here, and the repository is laid out well. Nice work!
Love the suggestion, I'll clone it down and start poking around.
I believe your intuition about layout experiments needing to be of different genres is correct. I think you could have a pretty wide range of debugging opportunities (imbalanced belts, inserters fighting for items, insufficient power at full load leading to throughput loss, etc) for the first. The second feels like it would be nicely encapsulated by focusing on optimizing for ratios, although seeing an agent realize that they can get more throughput by simply copy/pasting a block and upgrading a belt would be pretty wild (depending on the recipe, of course). Maybe nuclear power / heat exchanger ratios are a bit too far down the path, but optimizing for copper cable use in green circuits is pretty important and fairly early in the tech tree?
It seems like there are a lot of interesting experiments to be had here. The lab-play scenarios having a time-related component seems like a good idea, I assume most Factorio players that keep biters on treat them as a combined temporal-spatial constraint, so you have a sort-of proxy comparison to a real game situation when you put the agents on a timer.
I like the way that the framework design is testing different things than micromanagement proficiency, such as what we have seen in DOTA 2 or StarCraft 2 experiments. Notably, severe worker micromanagement (in the case of the latter game) becomes a way to squeak out extra minerals when you have infinite APM available. This is an interesting learned behavior in a narrow context, but that tactic is really control intensive and has a high chance for even pro players to screw it up when attempting to do so. It also doesn't seemingly give additional insight into an agent's longer-term planning, execution, and analytical performance. FLE seems way more interesting as a higher-level "thinking" evaluation framework, with all that in mind.
Any plans for layout optimization benchmarks? As in, start with a given factory cell with X inputs and Y outputs, and optimize its performance.
It sounds like your nephew has a project in mind. Start there with the basic dependencies and that will start laying out a competency roadmap which looks a lot like a curriculum.
Quick aside: Automate small/mid business manufacturing? Admirable, but will probably choke on the scale problem, so I think the journey will be vastly more interesting than the destination...which is good! Turns out there's a lot of robotics that can be broadly applied.
I fell into industrial work right out of undergrad as a EE, not intending to work in the rust belt or manufacturing or anything of the like. I erroneously assumed it was not important, not sexy, not interesting. How wrong that was.
How things are made is so important, not only for our society but also as learning experiences for engineers, planners, logicticians, and more. As a career roboticist, the time I spent in the manufacturing industry seems invaluable to me now.
I wonder if a mod that changes the graphics to a visual style shown in the link (I'm thinking the Carbot graphics [0] swap for StarCraft 2 in terms of scope) would make it feel more to your liking.
The visualizations are so similar to integrated circuit layouts; they immediately reminded me of some of the coasters that GamersNexus sell which represent simplified computer subsystems.
Having seen what some other university administrations have done to darling students and innovators even just around such things as lab space, parent comment is right on the money.
How much did you learn from the lessons of other contemporary robotics frameworks that are out there? Do you envision focusing in on particular types of tasks later, or is it still uncertain how your robot design will evolve as the dataset grows?
Another HN roboticist chiming in; the videos look good! I have many questions, but will keep it to just a couple for brevity's sake.
- How much are you able to use the robot's internals to estimate the gripped bag's inertial properties? If you're trying to put rigid, heavy bags below light, amorphous bags, are you adjusting final placement location on-the-fly?
- How dynamic is your scene beyond what we can observe? If you're using light curtains with a single robot on a track, and you're able to estimate some rough geometry of and track the bags down the conveyance, and you are updating occupancy of the bin as you're going, what else is there?
- Is this just inside for the foreseeable future or are you all going to tackle unpacking outside, as well as all the, ahem, baggage that comes along with operating outside the terminal walls.
Nice and straightforward problem, relatively speaking.
Background: Started in industrial automation (lots of Fanuc, Yaskawa, Omron, etc.), built a lot of cool systems with cool people that made things with robots. Pivoted to "general" robotics in grad school. Been spending the last 5+ years making "general" robots.
I think the best thing for learning robotics looks pretty similar to learning a programming language: Have a specific task in mind that the robot/programming language will help you solve. Even if its just a pick-and-place and a camera, or a shaker table with a camera over top, or a garden watering timer/relay combo. Just work on something specific with your toy robot and you'll naturally encounter much of the difficult things about robotics (spatial manipulation, control, timing, perception, drivers (GODDAMN DRIVERS), data, you name it).
Going right to a high-DOF arm or trained LLM is always cool, but the person who hacks together a camera/relay/antenna to automate some gardening task or throws some servos and slides together to make a Foosball robot is doing the most interesting things, in my opinion.
Because it is. There is an abundance of tutorials showing people how to get Gazebo simulations going, or set up a rudimentary classifier in Pytorch, or actuate motors with Arduino using whatever framework of the week (I even wrote some!).
The hardware has really made strides too, easy and cheap sensors, controllers, cameras. It's awesome how quickly someone can plug-and-play a Realsense with a servomotor or pneumatic slide and start manipulating the world.
The thing that's usually underappreciated is that once you understand how to code a robot, you are barely closer to having solved a practical problem. There are lots of practical problems in the world where spending 4 hours learning how to use $PERCEPTION_API would be actually better spent spending 4 hours understanding more about the widget being perceived or the object being manipulated. Getting into robotics has never been easier, getting something useful out of robotics is still the trick.