Tesla auto-pilot keeps confusing moon with traffic light then slowing down(reddit.com)
reddit.com
Tesla auto-pilot keeps confusing moon with traffic light then slowing down
https://www.reddit.com/r/SelfDrivingCars/comments/orz21z/tesla_autopilot_keeps_confusing_moon_with_traffic/
92 comments
Posted a week ago: https://news.ycombinator.com/item?id=27934539
I can't understand why instead of using all the possible senses: vision, auditory, radar, lidar they want to only use vision as if there's some sense of "purity" in doing that.
I think one of the main reasons for Tesla to use only vision is that then the mistakes the system makes will be more understandable and thus acceptable to humans. The way radar and lidar systems fail and cause wrecks will seem strange to humans, like running into stationary objects. Even if using these sensors would make system better overall, they might not be worth including if one is shooting for human acceptance of the system as fit for use.
An autocar seeing something that looks like a red light and breaking (or breaking for a bit and then continuing after deciding it was not a red light) is going to be acceptable to people because that is how they behave. Phantom breaking due to some radar or lidar input that the human can't perceive is going to be interpreted very negatively. I imaging auditory might be added at some point, if it is not already being used.
An autocar seeing something that looks like a red light and breaking (or breaking for a bit and then continuing after deciding it was not a red light) is going to be acceptable to people because that is how they behave. Phantom breaking due to some radar or lidar input that the human can't perceive is going to be interpreted very negatively. I imaging auditory might be added at some point, if it is not already being used.
I find the mistakes it makes to be less understandable because I can see with my eyes that the light in the sky is the moon, and not a traffic light. Why can’t the car?
Three reasons:
1) They have hundreds of millions of cars sold already with cameras that they can up-sell this to. That means more revenue. But more importantly, more training data. Massive amounts of it. Training data is the real value here. Radar/lidar/etc. might be able to detect better what is obvious to a human just looking at a thing. But given enough training data, a machine learning can probably replicate that capability so you don't actually need the fancy sensors. Adding new sensors to the mix would set them back quite a while on that front.
2) Simplicity. More sensors means more complexity integrating all the signals and gathering the right training data. More failure modes, etc. It probably also means more compute power needed to process all that data. More complex testing, etc. Scaling by keeping the sensor platform simple is a good move here.
3) The hard part of autonomous driving is actually interpreting visual signals in complex or unusual/rare situations. Roads are designed for humans with eyes. Lidar sees a blob, radar detects a pole, a camera sees a traffic sign, road markings that mean something, etc. It's a much richer signal. All the important stuff on roads is clearly visible. So, cameras are far more important for this than lidar/radar. Those are really great for avoiding crashing into things. Not so much for interpreting and classifying those things. And Tesla seems to be doing pretty OK with not crashing into things. Mostly, the amusing edge cases have to do with misinterpreting visual signals for which radar and lidar are probably not that relevant.
It's an interesting approach that they clearly believe that they can make work. It does not actually stop them from later adding more hardware to enhance things if they decide those things are needed. But it's quite interesting how far they are getting with just cameras.
1) They have hundreds of millions of cars sold already with cameras that they can up-sell this to. That means more revenue. But more importantly, more training data. Massive amounts of it. Training data is the real value here. Radar/lidar/etc. might be able to detect better what is obvious to a human just looking at a thing. But given enough training data, a machine learning can probably replicate that capability so you don't actually need the fancy sensors. Adding new sensors to the mix would set them back quite a while on that front.
2) Simplicity. More sensors means more complexity integrating all the signals and gathering the right training data. More failure modes, etc. It probably also means more compute power needed to process all that data. More complex testing, etc. Scaling by keeping the sensor platform simple is a good move here.
3) The hard part of autonomous driving is actually interpreting visual signals in complex or unusual/rare situations. Roads are designed for humans with eyes. Lidar sees a blob, radar detects a pole, a camera sees a traffic sign, road markings that mean something, etc. It's a much richer signal. All the important stuff on roads is clearly visible. So, cameras are far more important for this than lidar/radar. Those are really great for avoiding crashing into things. Not so much for interpreting and classifying those things. And Tesla seems to be doing pretty OK with not crashing into things. Mostly, the amusing edge cases have to do with misinterpreting visual signals for which radar and lidar are probably not that relevant.
It's an interesting approach that they clearly believe that they can make work. It does not actually stop them from later adding more hardware to enhance things if they decide those things are needed. But it's quite interesting how far they are getting with just cameras.
For #1: wouldn't more sensors be even better for training? If vision does not pick up an object like a human, but lidar detects a human sized object, the vision portion of the model can 'learn' from it's mistakes.
The other two sound like good reasons though
The other two sound like good reasons though
Except that would kind of invalidate all the training data they've gathered for the last years which does not have data for these sensors. That would be a major setback and might take years to fix. Probably they started ignoring the radar sensor content quite some time ago when they realized they did not need it. They never had any lidar.
Obstacle detection with lidar/radar is only interesting if you assume that object avoidance is a problem with camera based obstacle detection right now. There are lots of incidents with Teslas but I don't recall them running over pedestrians or crashing into vehicles a lot. Mostly incidents are about misinterpreting visual signals; not about crashing into stuff. If anything, their safety statistics are pretty good when autopilot is on. The cars still do dangerous/illegal/misguided things due to misinterpreting of traffic situations but it's then smart enough to get the driver out of trouble before bad things happen.
Obstacle detection with lidar/radar is only interesting if you assume that object avoidance is a problem with camera based obstacle detection right now. There are lots of incidents with Teslas but I don't recall them running over pedestrians or crashing into vehicles a lot. Mostly incidents are about misinterpreting visual signals; not about crashing into stuff. If anything, their safety statistics are pretty good when autopilot is on. The cars still do dangerous/illegal/misguided things due to misinterpreting of traffic situations but it's then smart enough to get the driver out of trouble before bad things happen.
Pure cost cutting at expense of safety certainly.
So they could put PS5-level gaming rigs in their cars? How logical https://www.theverge.com/2021/6/1/22462660/amd-tesla-model-x...
Oh you’re right, I didn’t realize they are investing in safety features like putting gaming consoles in car dashboards. That will save millions of lives.
All Hail Elon.
All Hail Elon.
Humans do fine with only 2 cameras
Chimpanzees have the same number of cameras as humans. Yet we don't let them drive. It's a puzzler.
/s
*And a brain that outclasses anything Tesla could currently put into their vehicles by numerous orders of magnitude.
Which of the above would solve this problem?
I'm not an expert, but I'm guessing Tesla's version of lidar wouldn't detect the moon. It's probably a bit out of range.
And when the moon shows up next to a pole, a bridge or any high up structure, the car comes to a stop because it "sensed" something. You must solve vision anyway, so why use a crutch?
[deleted]
LIDAR.
You'd be able to tell there's no object anywhere near the car and disregard the light.
You'd be able to tell there's no object anywhere near the car and disregard the light.
https://youtu.be/NSDTZQdo6H8?t=391
In the above video Andrej Karpathy claims that LIDAR required previously compiled maps, the creation of which is not scalable. Do you disagree with that point of view?
In the above video Andrej Karpathy claims that LIDAR required previously compiled maps, the creation of which is not scalable. Do you disagree with that point of view?
Andrej's claim isn't that you need those maps, it's that using those maps is how today's LIDAR approach work.
Without the maps, you'd need to rely pretty heavily on vision to do the right thing.
What LIDAR gives you is a very accurate point cloud to work from. You can get a point cloud from vision (and indeed, that's what telsa is doing) but it won't be near as accurate as LIDAR is by default.
Without the maps, you'd need to rely pretty heavily on vision to do the right thing.
What LIDAR gives you is a very accurate point cloud to work from. You can get a point cloud from vision (and indeed, that's what telsa is doing) but it won't be near as accurate as LIDAR is by default.
Unless it's hanging from a cable https://www.masterfile.com/image/en/700-08743685. But then, it could be a streetlight so only vision can solve this problem https://www.123rf.com/photo_2753272_hanging-street-lamp.html
You need vision to determine it is or isn't a traffic light.
Lidar would be able to tell you if the object should even be evaluated.
A light hanging from a cable would still get a lidar ping back. The moon would be a void with no lidar ping.
Lidar would be able to tell you if the object should even be evaluated.
A light hanging from a cable would still get a lidar ping back. The moon would be a void with no lidar ping.
So LIDAR says there's something, and maybe the car should stop.
Maybe is not acceptable (and blind in the rain/fog), so you must rely on vision to make the decision – which is why Tesla goes straight to the heart of the problem.
They haven't solved it yet, obviously, but neither has Waymo or anyone else. Tesla is probably collecting more images of 'moon or traffic light?' images than the rest of the industry combined already.
Maybe is not acceptable (and blind in the rain/fog), so you must rely on vision to make the decision – which is why Tesla goes straight to the heart of the problem.
They haven't solved it yet, obviously, but neither has Waymo or anyone else. Tesla is probably collecting more images of 'moon or traffic light?' images than the rest of the industry combined already.
Lidar could make this false prediction less likely.
That said, the computer vision is still far from human level. It lacks a common sense around how the world works.
That said, the computer vision is still far from human level. It lacks a common sense around how the world works.
[deleted]
Probably just a bit more training data and cameras will do just fine. It's a funny edge case; nothing more. I'd be surprised if it will remain a problem.
Radar and Lidar perhaps, being based on reflection of radio waves/laser off surfaces.
Radar isn't sensitive enough. LIDAR is.
LIDAR is so sensitive it can't reliably detect traffic lights in the rains. So you must rely on vision anyway (which is Tesla's).
I suspect both would have been able to tell that the moon isn't a traffic light.
[deleted]
It's entirely cost.
I mean, it's gotta be style and complexity too. LIDAR are quite ugly on test vehicles. Cameras are much simpler and easier to hide among the body.
Because Tesla lacks money?
Their automotive gross margin is 28% (Toyota is at 17.76%). If one automaker could afford LIDAR, it's Tesla.
Their automotive gross margin is 28% (Toyota is at 17.76%). If one automaker could afford LIDAR, it's Tesla.
LIDAR has come down a lot in cost over the last decade, but it would still eat up that whole 28% and more.
Using all the senses would not solve the problem. Vision is the only one with enough resolution to matter.
That said: - way cheaper - more reliable - less power hungry - simpler and easier to debug.
That said: - way cheaper - more reliable - less power hungry - simpler and easier to debug.
There's a possibly or partially apocryphal story about the moonrise causing a false alarm of a Russian ballistic missile attack from the BMEWS system in 1960 when they first turned it on in Thule. Anyway the idea that there are certain kinds of bugs which reoccur from one generation of system to another is amusing or sobering depending on the consequences.
http://taihendaro.cynic.net/2010/01/moon-as-soviet-missle-at...
http://taihendaro.cynic.net/2010/01/moon-as-soviet-missle-at...
This is a problem with learning the appropriate context. In the case of a yellow traffic light, there should be a traffic light fixture and two other (unlit) lights. Without this context, it becomes apparent the yellow pixels in a circular shape (representation to the network) are not a traffic light at all.
Tesla’s vision/ml systems are amazing. I would love to learn more about how unit testing for this type of error is done. Without some intermediate semantic representation, I don't see how these large, multi-head, end-to-end systems can isolate and regression? System tests are maybe possible, but it's unclear how well a system test would generalize to related, but unseen cases.
Tesla’s vision/ml systems are amazing. I would love to learn more about how unit testing for this type of error is done. Without some intermediate semantic representation, I don't see how these large, multi-head, end-to-end systems can isolate and regression? System tests are maybe possible, but it's unclear how well a system test would generalize to related, but unseen cases.
> Tesla’s vision/ml systems are amazing
More importantly, they're the only ones accessible to the average consumer. Waymo/Zoox/Daimler all have equally if not more impressive systems.
A real issue with tesla is that they want to be vision-only, which is going to make getting to level 5 first almost impossible.
BTW - knowing where the moon is happens to be an extremely solved problem.
More importantly, they're the only ones accessible to the average consumer. Waymo/Zoox/Daimler all have equally if not more impressive systems.
A real issue with tesla is that they want to be vision-only, which is going to make getting to level 5 first almost impossible.
BTW - knowing where the moon is happens to be an extremely solved problem.
> knowing where the moon is happens to be an extremely solved problem.
It's true... but I don't think that's how Tesla would solve it. Their goal is to create a neural network "driver" which can drive in any place even if it has never seen it before. They'd rather teach their neural network that the moon and stoplights are not to be confused visually. Thought I suppose in searching for training examples they could use the known position of the moon for approximate labeling.
It's true... but I don't think that's how Tesla would solve it. Their goal is to create a neural network "driver" which can drive in any place even if it has never seen it before. They'd rather teach their neural network that the moon and stoplights are not to be confused visually. Thought I suppose in searching for training examples they could use the known position of the moon for approximate labeling.
I would have looked at that as a stretch goal, and very stretching at it. Because I won't need very soon an autopilot to take me through the woods, especially if it's that trustworthy as it feels to be. I'd be happy to have one to drive reliably highway and wake me up when we're in close range from destination.
Well that’s what Tesla is doing. The current system can drive itself on highways and they are pushing to create a system capable of driving on all roads.
>which can drive in any place even if it has never seen it before
Isn't that impossible considering these networks need training and therefore have seen everything before?
Isn't that impossible considering these networks need training and therefore have seen everything before?
What I mean specifically is that competitors self driving systems use "HD maps" meaning they store the entire world in 3D and then they localize themselves to that world (at least, this is what Karpathy says competitors do). So those systems cannot navigate any stretch of road they haven't seen.
But with the Tesla, it is learning to drive in general. It does not need an HD map of a fork in the road to understand how to navigate it. Just as a person who learned to drive in California will have little trouble driving in Florida, a neural network that has learned to drive on a million intersections will be pretty good at navigating most intersections. Especially because the corner cases will stand out and become integrated in to training. So it may see many intersections, but it will generally know what to do with one even if it has never seen it before.
Though I would suspect that the competitors are perhaps using HD maps to jumpstart a system that long term would behave more like the Tesla one. Mapping every road is a lot to ask.
But with the Tesla, it is learning to drive in general. It does not need an HD map of a fork in the road to understand how to navigate it. Just as a person who learned to drive in California will have little trouble driving in Florida, a neural network that has learned to drive on a million intersections will be pretty good at navigating most intersections. Especially because the corner cases will stand out and become integrated in to training. So it may see many intersections, but it will generally know what to do with one even if it has never seen it before.
Though I would suspect that the competitors are perhaps using HD maps to jumpstart a system that long term would behave more like the Tesla one. Mapping every road is a lot to ask.
Complicated - they should be able to piece together a route with reasonably up to date "street view but like for self driving" imagery, up to date maps, and reasonable weather conditions.
One actually really important feature of these systems is how they handle failure. If the car gets confused, how does it handle it?
One actually really important feature of these systems is how they handle failure. If the car gets confused, how does it handle it?
> In the case of a yellow traffic light, there should be a traffic light fixture and two other (unlit) lights. Without this context, it becomes apparent the yellow pixels in a circular shape (representation to the network) are not a traffic light at all.
Permanent yield signals are often only one flashing yellow light. Crosswalk signals are often only flashing yellow lights when active. Temporary construction barrier signals are often only yellow flashing lights. Fire station signals often have only red and yellow lights, no green. Even when all three lights are present, they may also be oriented horizontally, or in triangular shapes.
Which isn't to say there isn't more context to learn from, but just about the only true unifying trait among all these indicators that you should perhaps pay attention and slow down is a bit of yellow light. It need not even be circular: yellow arrows are far from uncommon, including "straight ahead" yellow arrows for intersections where turns are forbidden.
Permanent yield signals are often only one flashing yellow light. Crosswalk signals are often only flashing yellow lights when active. Temporary construction barrier signals are often only yellow flashing lights. Fire station signals often have only red and yellow lights, no green. Even when all three lights are present, they may also be oriented horizontally, or in triangular shapes.
Which isn't to say there isn't more context to learn from, but just about the only true unifying trait among all these indicators that you should perhaps pay attention and slow down is a bit of yellow light. It need not even be circular: yellow arrows are far from uncommon, including "straight ahead" yellow arrows for intersections where turns are forbidden.
You can learn about their systems by watching talks by Andrej Karpathy. As a robotics engineer interested in vision, their architecture is inspiring. This talk [1] is a good overview but each talk he gives is a little different so search for more if you want to know as much as possible.
But the big thing is that their autonomy computer can be programmed to look for odd scenarios and send them back home. Tesla uses their fleet of hundreds of thousands of cars to collect edge cases like this, and then they have a kind of compartmentalized neural network system that breaks apart disparate tasks. With their collected examples they can create unit tests to ensure that the moon stops activating the stoplight detector. Once trained, the unit tests presumably help ensure they don't end up with future regressions.
So basically every time you see a Tesla do a weird thing, there is a good chance it will stop doing it soon enough. At least if it's hitting hacker news.
[1] https://www.youtube.com/watch?v=hx7BXih7zx8
But the big thing is that their autonomy computer can be programmed to look for odd scenarios and send them back home. Tesla uses their fleet of hundreds of thousands of cars to collect edge cases like this, and then they have a kind of compartmentalized neural network system that breaks apart disparate tasks. With their collected examples they can create unit tests to ensure that the moon stops activating the stoplight detector. Once trained, the unit tests presumably help ensure they don't end up with future regressions.
So basically every time you see a Tesla do a weird thing, there is a good chance it will stop doing it soon enough. At least if it's hitting hacker news.
[1] https://www.youtube.com/watch?v=hx7BXih7zx8
But in foggy or dark situations, a yellow light can indeed appear as a floating yellow circle.
There was another video of a Tesla being confused by a truck carrying traffic lights on the truck bed. I do like the idea of what Waymo does, which is to run their model through a virtual world to see how it performs. It's amusing to imagine each of these edge cases being added in as they are encountered.
I've definitely driven through places where there is only one light. Blinks red for stop (no stop sign). Many times, for cross traffic you'll have the one light but blinking yellow as a warning at the intersection.
If that would be a traffic light in front of the car, it would get bigger as a car moves towards it. Basic perspective knowledge. But it seems the system is not considering it or not putting enough weight on it.
Karpathy's latest talk says they have 6000 tests cases(video clips) that each new version of the model has to predict the right answer on, for it to be released.
Is it slowing down because it is going 65 on a 55mph zone?
So if I buy a truck and paint a big traffic light on its back and maybe even put a red light there I will be fooling all Teslas? Count me in!
Edit: this also reminds me of those Road Runner episodes where Road Runner paints a rock wall like it's the continuation of the road so that Coyote can smack his head on it.
Edit: this also reminds me of those Road Runner episodes where Road Runner paints a rock wall like it's the continuation of the road so that Coyote can smack his head on it.
An even harder case: https://futurism.com/the-byte/tesla-autopilot-bamboozled-tru...
I only have two cameras and have never confused the moon with traffic lights.
My three-year-old has mistaken a streetlight seen through her bedroom window for the moon more than once, but then I don’t let her drive either.
EDIT: Though even that isn't a traffic light, and not the other way around.
EDIT: Though even that isn't a traffic light, and not the other way around.
If your head was in a vice and your eyeballs were superglued into the straight-ahead position you might have more trouble.
That's not a great comparison, since our focal area is different than that of cameras. I'm pretty sure anyone looking at the camera feeds in a Tesla wouldn't be confused either; this is a software problem.
So tesla just needs to make the cameras on their cars swivel around, Thomas the tank engine style?
Several European countries have a white or yellow reflective border around traffic lights to help with recognition, especially for those who might be color-blind: with a bright rectangle, it's easier to see the position of which light is lit, rather that only rely on the color.
A couple decades later, I'm now noticing some traffic lights in the US starting to gain this feature. I'm sure it will help with machines trying to parse the traffic light state... and also humans.
A couple decades later, I'm now noticing some traffic lights in the US starting to gain this feature. I'm sure it will help with machines trying to parse the traffic light state... and also humans.
Use lidar, goddamnit.
lidar isn’t able to read the state of traffic lights as far as I know, and I believe lidar based systems detect the location of traffic lights from maps
Lidar cannot identify the state of a traffic light but it can confirm and understand its physical position.
Lidar can't tell a streetlight from a traffic light, so this has to be resolved by vision anyway.
Yeah. But that's not what the OG comment is suggesting. The use of Lidar here would rule out a light resembling a traffic signal by virtue of there being no laser reflection of that object: lidar would complement vision to rule out false positives.
They look different so it should be able to tell them apart if the model has been trained on these shapes. My robot vacuum uses lidar, the maps it makes of my apartment when it cleans are remarkably accurate where you can identify certain objects pretty clearly.
If you overlay the point cloud with the image you're interpreting, it should become apparent that the "yellow light" has no accompanying red or green lamps, no pole holding it, no shroud around it, and is at some distance exceeding the depth limits of your lidar.
I wonder if the concern is that LiDAR looks "ugly" or nerdy on the front of the car. And a bit more aero drag.
Elon can't afford to put LIDAR on cars and sell them to a large audience. It's still too expensive. So instead, he makes excuses for why they don't do it while rolling out objectively worse perception systems.
Lidar can be embedded in the headlights or on the rear view mirror; there is no need for ugly protrusions.
Yeah, then they could never ever ship like every other Lidar company
Why it doesn't just calculate moon's position given current date, time of day and coordinates?
Yeah, this is what I do when I drive. Otherwise I think it's a traffic light and I panic.
:) That's exactly what Teslas do currently.
And what if the moon is not visible but there's a traffic light where the moon should be?
I get that this is a problem, and if I paid 10 grand for this feature I would be pissed too if it put me in danger. But it also seems that compared with many things they are working on a more solvable problem with more data.
That does kind of look like a traffic light in haze or fog
```recalculating route...driving to: moon```
The moon appears white/grey. Traffic lights are red, green, or yellow. Seems easy enough to make a distinction. Additionally the relative size of the moon in the sky does not change.
> The moon appears white/grey.
Not all the time. I've seen reddish/orangeish moons, especially if it's something near the horizon near sunset or sunrise. It's also reddish during lunar eclipses.
Not all the time. I've seen reddish/orangeish moons, especially if it's something near the horizon near sunset or sunrise. It's also reddish during lunar eclipses.