Tests show all driver assist systems can be fooled(jalopnik.com)
jalopnik.com
Tests show all driver assist systems can be fooled
https://jalopnik.com/tests-show-all-driver-assist-systems-can-be-fooled-not-1847474485
64 comments
> Level 2 is not intended to be autopilot.
Then Tesla should change the name and offer refunds to anyone who bought Full Self Driving.
Then Tesla should change the name and offer refunds to anyone who bought Full Self Driving.
I 100% agree with you
yes, and they should be sued for false advertising
babesh(2)
> That's all the incentive you need, and it's the same incentive we have for people driving today.
Arguably, the incentives today aren't strong enough given the level of pedestrian fatalities
Arguably, the incentives today aren't strong enough given the level of pedestrian fatalities
I mean obviously, that's why people buy them. Everybody thinks they're an incredibly safe driver and most of them privately think the safety aspect is the smallest reason they buy these systems. They get bought because people want to text and watch Netflix on the way to work.
So I believe there's a tradeoff here. People drive to work all the time staring at their phones and only glancing up at the road every few seconds. They always have the option of turning the assist systems off and going back to distracted driving the old fashioned way. Obviously companies have no financial incentive to do this, but I posit that not making it impossible to use assist systems while completely distracted is actually a net benefit for road safety.
I'd rather have Honda Sense driving a car down the interstate by itself than some dude who's only looking up when the road curves.
So I believe there's a tradeoff here. People drive to work all the time staring at their phones and only glancing up at the road every few seconds. They always have the option of turning the assist systems off and going back to distracted driving the old fashioned way. Obviously companies have no financial incentive to do this, but I posit that not making it impossible to use assist systems while completely distracted is actually a net benefit for road safety.
I'd rather have Honda Sense driving a car down the interstate by itself than some dude who's only looking up when the road curves.
> People drive to work all the time staring at their phones and only glancing up at the road every few seconds.
Do you know people who drive like this? Driving this way is (a) very illegal and (b) recklessly endangering other people. I doubt it's especially common, at least in most of the US.
Do you know people who drive like this? Driving this way is (a) very illegal and (b) recklessly endangering other people. I doubt it's especially common, at least in most of the US.
I question this as well. A citation would certainly be helpful.
Are we even living in the same world? I see it every day.
I don't know how common it is, but people do drive like this. From a 2012 news article [1] about Rob Ford, the mayor of Toronto at the time:
Reporter: "Sir, there's a picture that went out on Twitter this morning of you reading while still driving on the Gardiner [Expressway]."
Ford: "Yeah, probably. I'm busy."
Reporter: "So you read while driving?"
Ford: "Yeah, probably, yeah. I'm try[ing] to catch up on my work and you know I keep my eyes on the road, but I'm a busy man."
In October, Ford was accused of illegally dialing numbers on his cellphone and talking on it as he steered his gold minivan westbound along Dundas Street West near Spadina Avenue.
I once saw a guy playing the flute while he was driving in Austin, presumably with his knees. Couldn't believe it.
I've had it show up in my own family when I noticed my Plex server was in use while I knew the person using it was on the road. Needless to say that resulted in an angry discussion.
You obviously don’t own a Tesla. This isn’t possible.
This article is contradicting you and specifically linking to this other article in the first paragraph as proof: https://jalopnik.com/consumer-reports-shows-how-easy-it-is-t...
I have a very different definition of "easy" compared to Consumer Reports.
> Fisher next placed a small, weighted chain on the steering wheel, to simulate the weight of a driver’s hand, and slid over into the front passenger seat without opening any of the vehicle’s doors, because that would disengage Autopilot. Using the same steering wheel dial, which controls multiple functions in addition to Autopilot’s speed, Fisher reached over and was able to accelerate the vehicle from a full stop.
Yes, the good old easy method of putting a weighted chain to the steering wheel and climbing over to the passenger seat.
> Fisher next placed a small, weighted chain on the steering wheel, to simulate the weight of a driver’s hand, and slid over into the front passenger seat without opening any of the vehicle’s doors, because that would disengage Autopilot. Using the same steering wheel dial, which controls multiple functions in addition to Autopilot’s speed, Fisher reached over and was able to accelerate the vehicle from a full stop.
Yes, the good old easy method of putting a weighted chain to the steering wheel and climbing over to the passenger seat.
Agreed, but Tesla should still have a weight sensor on seat to prevent this nonsense.
...and a camera for face tracking and maybe ask some basic trivia questions from the driver randomly? :D
The problem with building idiot-proof things is that idiots are way too inventive.
The problem with building idiot-proof things is that idiots are way too inventive.
Why? The only people doing this are “journalists” seeking clicks (or attacking Tesla on behalf of unions or unionized auto makers) or TikTokkers making fake videos.
The seat sensor wouldn’t add any extra inconvenience, and they already have the technology. So it seems kind it could only help.
It’s not possible to watch Netflix on your Tesla screen when the car is not in park. If you want to construct a Rube Goldberg device on your steering wheel so you can watch Netflix on your iPhone in the passenger seat, you have bigger problems than crashing.
[deleted]
Isn't this obvious?
"tests show human drivers can be fooled too"
"tests show human drivers can be fooled too"
I still don't understand the entire premise here. Who cares if a driver can deliberately defeat these things? That's not the threat model!
A driver nag is an attentiveness reminder. You're driving a car. You're expected to be paying attention. No one else is responsible for safety. So the car wants (for both safety and liability reasons) to make sure you're doing what you're supposed to do. And it works. The car nags me and I tug the wheel to prove I'm there. Most of the time I'm paying attention anyway, but sometimes it's because I got distracted. It works.
If a driver deliberately wants to behave unsafely, there's no way to stop that, because there's an infinite number of ways drivers can behave unsafely. Do we want a system to detect when drivers speed up on yellow? Tesla could absolutely implement that. Why aren't we demanding it? Prevent drivers from approaching cars in adjacent lanes with higher than a 30mph speed delta? Could do that too. Prevent accelerating in the wrong direction on a road? Possible.
But no, all that stuff would be dumb, because it doesn't correspond to an actual safety problem with the hardware. We recognize it's driver behavior at fault, and we don't want our cars regulating behavior.
Why is this different?
I submit it's different because "Actually Elon Musk is an Asshole and TSLA went to the moon". Which isn't a very good premise for automobile safety designs.
A driver nag is an attentiveness reminder. You're driving a car. You're expected to be paying attention. No one else is responsible for safety. So the car wants (for both safety and liability reasons) to make sure you're doing what you're supposed to do. And it works. The car nags me and I tug the wheel to prove I'm there. Most of the time I'm paying attention anyway, but sometimes it's because I got distracted. It works.
If a driver deliberately wants to behave unsafely, there's no way to stop that, because there's an infinite number of ways drivers can behave unsafely. Do we want a system to detect when drivers speed up on yellow? Tesla could absolutely implement that. Why aren't we demanding it? Prevent drivers from approaching cars in adjacent lanes with higher than a 30mph speed delta? Could do that too. Prevent accelerating in the wrong direction on a road? Possible.
But no, all that stuff would be dumb, because it doesn't correspond to an actual safety problem with the hardware. We recognize it's driver behavior at fault, and we don't want our cars regulating behavior.
Why is this different?
I submit it's different because "Actually Elon Musk is an Asshole and TSLA went to the moon". Which isn't a very good premise for automobile safety designs.
> Who cares if a driver can deliberately defeat these things? That's not the threat model!
Faking a "malfunction" and then suing could be part of the threat model to the car maker and/or the insurance companies that provide liability coverage.
Faking a "malfunction" and then suing could be part of the threat model to the car maker and/or the insurance companies that provide liability coverage.
Priuses had a bunch of fake "stuck accelerator" incidents a while ago.
> Why is this different?
Because the big shiny text that people pay $10,000 for says "Autopilot". Between the name and the large price tag, it conveys that it's higher than L2 autonomy. Putting in the fine print that it's not, doesn't change things.
Because the big shiny text that people pay $10,000 for says "Autopilot". Between the name and the large price tag, it conveys that it's higher than L2 autonomy. Putting in the fine print that it's not, doesn't change things.
Who buys Full Self Driving for 10k and thinks you can’t pay attention? The only people I ever see say that are people that don’t have it.
> Who buys Full Self Driving for 10k and thinks you can’t pay attention?
People who understand how English works? Like, if I spend 10k on something that is "Full Self Driving", I'm paying for the ability for it to drive itself, fully. That is to not watch the road. It's right there in the name. Otherwise call it "Kinda Self Driving"
But given the accidents where people turn on FSD and go to sleep, clearly not everyone thinks that. And the better FSD is, the more crashes are avoided and the higher that percentage is.
People who understand how English works? Like, if I spend 10k on something that is "Full Self Driving", I'm paying for the ability for it to drive itself, fully. That is to not watch the road. It's right there in the name. Otherwise call it "Kinda Self Driving"
But given the accidents where people turn on FSD and go to sleep, clearly not everyone thinks that. And the better FSD is, the more crashes are avoided and the higher that percentage is.
I understand that it's a proper name and that in English I shouldn't read too much meaning into proper names. The Big Mac isn't all that big. The Nike Dunk didn't let me dunk basketballs as a kid, but it was a cool shoe. One of my favorites was the ole Six Dollar Burger that by the end was costing you more than $6.
When you buy it its not just the name and a price.
First of all it informs you that it is not jet complete. There is a complete list of what features it has and what features it doesn't have.
You very clearly buying the right to have access to the newest features as the role out. This is very clearly spelled out when you buy it.
> But given the accidents where people turn on FSD and go to sleep
Do you have any actual data about people deliberately go to sleep?
I have seen no numbers what so ever to confirm what you are saying.
First of all it informs you that it is not jet complete. There is a complete list of what features it has and what features it doesn't have.
You very clearly buying the right to have access to the newest features as the role out. This is very clearly spelled out when you buy it.
> But given the accidents where people turn on FSD and go to sleep
Do you have any actual data about people deliberately go to sleep?
I have seen no numbers what so ever to confirm what you are saying.
> You very clearly buying the right to have access to the newest features as the role out. This is very clearly spelled out when you buy it.
All I know is in my personal experience, everyone who bought FSD feels conned. Maybe you went into it eyes wide open, and if so good for you. Maybe they expected a faster rollout of features, maybe they didn't notice that part of the sales pitch at all.
How big is the disclaimer when you buy FSD? I'm not sure that any disclaimer is really worth the horribly misleading name.
>Do you have any actual data about people deliberately go to sleep?
Well, there was the one where someone went into the back seat to sleep. And there were several other high-profile crashes where people turned on FSD and passed out at the wheel at over 100MPH.
And, we only have data on when the FSD fails, and crashes ensue. Heck, we probably only have data on fatalities because the news doesn't want to get sued.
All I know is in my personal experience, everyone who bought FSD feels conned. Maybe you went into it eyes wide open, and if so good for you. Maybe they expected a faster rollout of features, maybe they didn't notice that part of the sales pitch at all.
How big is the disclaimer when you buy FSD? I'm not sure that any disclaimer is really worth the horribly misleading name.
>Do you have any actual data about people deliberately go to sleep?
Well, there was the one where someone went into the back seat to sleep. And there were several other high-profile crashes where people turned on FSD and passed out at the wheel at over 100MPH.
And, we only have data on when the FSD fails, and crashes ensue. Heck, we probably only have data on fatalities because the news doesn't want to get sued.
There were 17 accidents with Autopilot and non with FSD. One of those was a death where somebody fell asleep on the highway.
There are literally 100s of videos of the Autopilot doing an avoiding maneuver that prevented potentially accidents, some potentially deadly.
> Maybe they expected a faster rollout of features, maybe they didn't notice that part of the sales pitch at all.
I think its fair to say you expected the features faster, but the claim its not spelled out is false. If people don't read what is very clearly written on the website when you buy it, then that is their fault.
There are literally 100s of videos of the Autopilot doing an avoiding maneuver that prevented potentially accidents, some potentially deadly.
> Maybe they expected a faster rollout of features, maybe they didn't notice that part of the sales pitch at all.
I think its fair to say you expected the features faster, but the claim its not spelled out is false. If people don't read what is very clearly written on the website when you buy it, then that is their fault.
Which is the most ridiculously meta argument. I have people all over the internet (amusingly never in real life) tell me to my virtual face that I clearly don't understand the product that I paid ten thousand dollars to buy, and that it doesn't do what I know damn well it does.
But I do, I did, and it does. So there.
But I do, I did, and it does. So there.
I dont get it. do you confirm the argument that buyers of tesla autopilot understand its an advanced driver assist system which at this time requires drivers attention? or you do not?
asking becuase the message yours responds to says exactly that: people whoy actually buy autopilot do know very well that it is a system that requests and sometimes even requires driver's attention.
so if you confirm that, then how is a meta argument, ridiculously so?
asking becuase the message yours responds to says exactly that: people whoy actually buy autopilot do know very well that it is a system that requests and sometimes even requires driver's attention.
so if you confirm that, then how is a meta argument, ridiculously so?
Who cares if a driver can deliberately defeat these things?
This is really to do with the expectations. Consider the safety belt, there is an expectation that with the safety belt on, the driver will not go through the windscreen in case of an accident. It is not the case that the driver has defeated the seat belt feature by not using the seat belt. A reasonable person will not have the expectation of having the safety provided by seatbelt without put it on.
Things are not as clear cut for assisted driving as can be seen in this thread.
This is really to do with the expectations. Consider the safety belt, there is an expectation that with the safety belt on, the driver will not go through the windscreen in case of an accident. It is not the case that the driver has defeated the seat belt feature by not using the seat belt. A reasonable person will not have the expectation of having the safety provided by seatbelt without put it on.
Things are not as clear cut for assisted driving as can be seen in this thread.
> I still don't understand the entire premise here.
It's because the premise is so weak. I can put a brick on the gas pedal, so no car is safe!
It's because the premise is so weak. I can put a brick on the gas pedal, so no car is safe!
Not to mention cruise control has the same "vulnerability": https://www.youtube.com/watch?v=f1q0FihkTfk
Is there any evidence that drivers are abusing Autopilot etc. any more than cruise control and causing a higher rate of accidents due to inattentiveness?
I was especially disappointed in Consumer Reports, which insinuated that these methods of "fooling" Autopilot were somehow connected to a fatal crash in Texas. https://www.consumerreports.org/autonomous-driving/cr-engine...
You have to get 11 paragraphs in before you get to this:
> Our evaluation does not provide specific insight into the Texas crash, but safety advocates and researchers at CR say that it does show that driver monitoring systems need to work harder to keep drivers from using the systems in foreseeably dangerous ways.
By this standard, cruise control should never have launched in any vehicle until technology was capable of verifying driver attentiveness with eye tracking.
Is there any evidence that drivers are abusing Autopilot etc. any more than cruise control and causing a higher rate of accidents due to inattentiveness?
I was especially disappointed in Consumer Reports, which insinuated that these methods of "fooling" Autopilot were somehow connected to a fatal crash in Texas. https://www.consumerreports.org/autonomous-driving/cr-engine...
You have to get 11 paragraphs in before you get to this:
> Our evaluation does not provide specific insight into the Texas crash, but safety advocates and researchers at CR say that it does show that driver monitoring systems need to work harder to keep drivers from using the systems in foreseeably dangerous ways.
By this standard, cruise control should never have launched in any vehicle until technology was capable of verifying driver attentiveness with eye tracking.
Don't say that too loudly, carpets became just such a Boogeyman not long ago
original non blog spam article:
https://www.caranddriver.com/news/a37260363/driver-assist-sy...
https://www.caranddriver.com/news/a37260363/driver-assist-sy...
No mention of comma.ai
So the summary is that humans can still outsmart current technology that monitors drivers attention. Not bad for humans.
But declaring Level 2 as garbage is stretching it. I am driving Level 2 (ACC plus lane assists plus speed sings recognition plus a few other things) for a few years now and I definitely get less tired on long drives. I recently drove coast to coast and it was more boring than tired because of the Level 2.
But declaring Level 2 as garbage is stretching it. I am driving Level 2 (ACC plus lane assists plus speed sings recognition plus a few other things) for a few years now and I definitely get less tired on long drives. I recently drove coast to coast and it was more boring than tired because of the Level 2.
How do you cope with the temptation to distract yourself?
As another "level 2 driver" with the same features as OP has, I don't understand how someone could even begin to think it would be safe to "distract themselves" at this early level of automation. The temptation to stay alive wins over any temptation to distract myself.
When I hear reports of accidents in Teslas because the driver "took a nap" or watched a movie or whatever, it's hard to feel empathy, because that's just reckless driving at this level.
Of course Tesla is to blame when they marketed, and maybe are still marketing, level 2 capabilities as being on par with an "Auto Pilot".
When I hear reports of accidents in Teslas because the driver "took a nap" or watched a movie or whatever, it's hard to feel empathy, because that's just reckless driving at this level.
Of course Tesla is to blame when they marketed, and maybe are still marketing, level 2 capabilities as being on par with an "Auto Pilot".
From Waymo's tests, people stop paying attention even when they know they're being monitored.
Because I know it's just an "assist". Basically the camera sees the white lines in the road and tries to bounce between them while keeping a safe distance to the car in front with a radar.
My car also requires me to put actual pressure on the wheel at all times, I think a 10-15 second lapse is allowed, then it starts to beep.
We do have the anecdotal stories of people buying RVs with "Cruise control", enabling it on a freeway and going to the back to make a cup of coffee. Then being all surprised_pikachu.jpg when they're being dug out from the wreckage.
My car also requires me to put actual pressure on the wheel at all times, I think a 10-15 second lapse is allowed, then it starts to beep.
We do have the anecdotal stories of people buying RVs with "Cruise control", enabling it on a freeway and going to the back to make a cup of coffee. Then being all surprised_pikachu.jpg when they're being dug out from the wreckage.
Every thirty seconds, the Tesla demands attention. If you're distracted, it beeps like crazy and a few times of that, the system disabled itself for the drive.
I also don't "feel" like it can drive itself so paying attention still is warranted.
Rather than saying Level 2 is garbage, we should look at the stats and determine whether it is or is not safer than driving without automation. A perfectly attentive driver will beat automation, but as all the annual deaths while driving show, that is certainly not how universally drive.
I also don't "feel" like it can drive itself so paying attention still is warranted.
Rather than saying Level 2 is garbage, we should look at the stats and determine whether it is or is not safer than driving without automation. A perfectly attentive driver will beat automation, but as all the annual deaths while driving show, that is certainly not how universally drive.
I plead the Fifth ;)
So what? This is the most absurd line of inquiry. Did you know that seat belts don’t work if you don’t put them on? The question is not can something be circumvented. The question is if the system is safe. There is no evidence presented here to suggest driver assist systems are a net hazard.
The truth is that some car manufacturers actually request it to be somewhat fool proof, but very few.
It costs hardware to run and cost to develop/test, and car manufacturing is all about price.
But also:
Today the algorithm knows the 3D geometry of your head, and knows that it does not makes sense with a flat face. But it is more important to get it to work good on everyone than to catch foolers.
Disclaimer: i'm one of all the developers behind camera monitoring.
It costs hardware to run and cost to develop/test, and car manufacturing is all about price.
But also:
Today the algorithm knows the 3D geometry of your head, and knows that it does not makes sense with a flat face. But it is more important to get it to work good on everyone than to catch foolers.
Disclaimer: i'm one of all the developers behind camera monitoring.
"It's currently the only system that allows hands-free driving for an extended period of time."
This is false. Comma.ai/OpenPilot allows this. Video seems under-researched.
This is false. Comma.ai/OpenPilot allows this. Video seems under-researched.
Level 2 is not intended to be autopilot. It's intended to be safer than driver unassisted.