Interesting post and great reference to [1] about why laundry hits a sweet spot of capability.
Interestingly the repeated critiques in the article are about sensor richness: primarily force feedback and tactility, which indicates lacking hardware. Software only robotics has a long and fraught history, but it really feels to me that current industrial hardware could be driven more intelligently without much change. No doubt the "ideal" robot for any given task requires developments in both.
I'm also curious about safety, since generally capable mechanisms need a multilayered safety stack that includes semantics, and cobot certification is likely not enough anymore. Examples: feeding someone the wrong pill, pouring a glass of water into electronics, cutting vegetables vs fingers.
Two of the authors of the original Aurora system left Microsoft to found https://silurian.ai/ - interesting to keep tabs on if you're interested in this space!
Thanks, Benjie! Great to see you here. I hope it's OK if I plug your excellent writings on robotics that I think everyone should check out: https://generalrobots.substack.com/
Great questions and a few answers to various parts of your question that you've mostly identified yourself I think:
- Golf clubs specifically (and most sporting equipment) actually goes down a different pipeline in most airports, since generally this stuff doesn't behave well on conveyor belts.
- Data driven approaches can tell you a lot just from visual information, usually about deformability of objects, but also about expected centers of mass, etc
- Part of the reason we're using single big robots is because you can use heavy duty end-effectors - grasp all the way around the object with a grasp that's predicted to be robust to these kinds of perturbances, and then use quick feedback to safely execute a plan to place it
You're absolutely correct though, that there's a long tail of things coming through and that some objects are very, very difficult. Our problem formulation then becomes identifying confidence in graspability, and deciding explicitly that we shouldn't attempt to grasp some object and should instead flag them for human handling.
Not a pest at all and I've long been frustrated with ROS - our early demos were actually just a single C++ binary with multiple threads running for perception, control, and visualization, and I byte-packed robot control messages in our own software to avoid using ROS.
Unfortunately this breaks down in a few ways that you're probably familiar with, given that you asked this question:
- A crash in (e.g.) a third party sensor driver can bring down your whole binary, any signal catching here is awkward and you end up wanting process isolation
- Perception is, for better, or worse, easiest to prototype and try off the shelf in Python / Pytorch, so you either end up with pybind11 and driving things in Python, ONNX which is IME brittle for some of the crazier Pytorch modules, or message serialization and process isolation.
ROS/ROS2 does _way_ too much in my opinion - why does it have a build system and a ton of packages? This plus pinning OS versions are huge pain points. Unfortunately I also think many community-contributed ROS/2 packages are fairly low code-quality, with notable exceptions. Overall, I'd prefer to have ROS be a pubsub library with a few extra tools for logging and visualization.
In the end, we're currently using ROS2 for the reasons listed above and for easy prototyping, but I'd like to move to something more like FPrime, Basis, Cerulion, or Copper in the near future. I really want to grow something in-house with Zenoh or IceOryx2, but don't want to waste a lot of time on middleware, since I don't think it's what's kept the problem from being solved.
(At the end of this post I now see you're working on Basis, I apologize that I'm over-explaining to you. I'd love to chat about Basis if you have some time in the next few days!)
There have been some solid attempts in this space before - many projects take on the whole baggage system design and end up very very complex and often over budget. We're focusing on introducing tech that plays well in a larger system, particularly in "brownfield" existing processes - our bet is that recent advances in robot autonomy give us ability to handle items that weren't possible before, and therefore our units can be introduced in a more flexible way.
John B was obviously aware from previous experience what a manual and injury prone process this was, but I've also been really surprised as I've dived deeper into airport operations myself.
Bagroom is definitely what we're targeting first - being indoors (usually) is a huge plus, and lets us focus on the manipulation part of the problem without going fully mobile yet.
That said, we're definitely targeting tarmac/ramp operations, particularly between a TUG/PowerStow and narrow-body bag carts. Inside the bin is much trickier but we agree it's the least ergonomic part of the job, you just can't move a massive industrial arm in and out of a plane very easily. We have it on our longer-term roadmap, though, and intend to leverage the baggage dynamics data we collect everywhere else to give us a head start on the packing and manipulation problems there, just with a different mechanism.
Cargo packing is a huge area of interest for us! Particularly around optimizing weight distribution in loaded planes, or just optimizing packing efficiency in general.
Also good question and something we've thought about. The difficulty there is actually getting the forklift tines out after placement. Actual forklifts in real warehouses rely on pallets as an affordance for manipulation, and we don't have that luxury here.
There have been some neat attempts with short conveyor belts as end effector tools [1]! Generally these systems rely on being able to rearchitect a significant amount of the process (building a controllable conveyor belt or rearchitecting part of the bag-room), and we're focused on dropping into existing processes.
We're really focused on health and safety aspects of this job - in a repetitive stress sense, these jobs are much more dangerous than many people imagine they would be and people end up with lifelong injuries.
Generally, regulators seem to be moving in this direction as well. The EU has introduced new regulations on the total amount of weight someone can move in a shift, and the Dutch government has mandated that baggage handling move away from manual processes like this in the near future.
Suction has gotten us pretty far at the prototype level but definitely isn't enough - we're testing out some new gripper designs that use suction as a broader part of an overall grasping system.
For these videos we have lidars and two Intel Realsense depth cameras mounted to the safety cage and on a wall. We're working on moving as many sensor on-robot as possible in the near future to aid with deployability.
The cup has taken us very far, which we're excited about, but it's definitely not enough - we're currently testing a multimodal claw-ish + suction gripper, which we've had good results with so far but aren't ready to unveil.
The teleop data is really useful for training data indeed, and lets us collect data on current failure points (e.g. with suction, just how far can we tilt this fabric bag before it peels away, etc). We're not going full behavior-cloned end-to-end for a lot of reasons (sample complexity, safety, adaptability, etc), but we do a lot of learning in specific parts of the system (particularly around grasping and placement).
The robot is indeed beefy, as many robots rated for 50kg applications are (check them out online). We've accidentally stress tested this unit way beyond 50kg without a hiccup, so we're very interested in figuring out what the right-size unit is for our application. There are a few other great aspects to this unit - it's a 7-DOF arm + 1 more DOF for the linear rail, so we have two extra degrees of freedom to play with for collision avoidance during planning.
Teleop and monitoring are systems that we've built ourselves and are pretty happy with. Since we use MuJoCo for simulation/visualization and some kinematics subroutines, to visualize, I just keep the MuJoCo GL context open after rendering and then throw all of our sensor data into it - it's very performant and low latency!
We've since introduced a message-bus layer that makes it possible to do it all over the internet etc, but adds the associated serialization and transport latency.
Operational changes for airlines are quite tricky - one of our bets is that most of the value for customers here is in handling "brownfield" deployments where you drop into an existing process, and that intelligence (or at least, good perception and reactive planning) really unlocks this ability from the robotics side.
For widebody planes, bags are already loaded into Unit Load Devices (ULDs), which are large semi-truncated boxes that get loaded directly onto aircraft. Narrow body planes don't use these (apparently) because they impact turn-time and decrease the amount of time a plane can be in the air, and also impacts how quickly bags come out, since it adds an extra step to unloading.
Many airport conveyance systems also load each bag into a bin, but those bins aren't loaded into the airplane because they belong to the airport and waste space/weight.
The best case for us would be a customer process change where everyone loads their luggage into perfectly regular and very sturdy hardshells, but this one's probably out of our hands.
Yep! We're obviously operating alongside a very well established world of palletization and other order fulfillment type robots.
We think that due to irregularity that it's not an easy tech transfer from the existing logistics world into the aviation world. We're very interesting in looking the other direction, though!
Solid question and something we think about a lot! Worst case is a weak zipper or similar. We're bringing a new gripper online which is more multimodal - some mechanical grasping, some suction, and the ability to choose what you use. We're moving away from pure suction partly for this reason and partly for textiles.
Suction is great though, and ~75% of bags checked through the US are hardshells, so it's something we're not ready to ignore entirely.
Good video! The overall question here is the blended rate of bags placed per minute, rather than how fast each action needs to be.
That said, the arm itself can move 180deg/s in every joint (roughly 5m/s max at the end effector) - these videos are still very much v1 and we're looking forward to leveraging more of the mechanical capability with a better gripper, better perception, and some new planning techniques we're rolling out in the next few months.
There are a lot of constraints in our planning, including what actions we can do for a particular bag/gripper interaction. Multiple robots working in collision range of each other isn't on the roadmap right now, but it's always a possibility.
Suction works really well but it's not enough, we're rolling out a new gripper soon that covers more cases mechanically (there's a very long tail in the distribution of what comes through airports).
Great point and we 100% agree! We have a new multi-modal gripper that we're testing now but are keeping hush while we bring it up. (We're actually swapping out a new arm too for a variety of reasons)
These videos are more to set a scene for where we're operating the the general process.
Interestingly the repeated critiques in the article are about sensor richness: primarily force feedback and tactility, which indicates lacking hardware. Software only robotics has a long and fraught history, but it really feels to me that current industrial hardware could be driven more intelligently without much change. No doubt the "ideal" robot for any given task requires developments in both.
I'm also curious about safety, since generally capable mechanisms need a multilayered safety stack that includes semantics, and cobot certification is likely not enough anymore. Examples: feeding someone the wrong pill, pouring a glass of water into electronics, cutting vegetables vs fingers.
[1] https://substack.com/redirect/82d94852-76b6-4b0d-8595-86e46a...