I agree, I think we are farther off from truly autonomous driving than we believe, too many factors play into it that tend to be forgotten( I'm guilty of it as well when it comes to self-driving or a lot of technologies I'd love to see sooner than later):
1. Some people just enjoy driving their cars, tinkering with them from an automotive perspective and the like, self-driving cars will inhibit that due to their heavier reliance on software, design differences and lack of modification abilities as opposed to today's vehicles, self-driving cars won't gain acceptance from that crowd for a while.
2. The first accident that is caused by a self-driving car will cause a major set-back in the eye of the public, airplanes are over-engineered for the sake of redundancy and safety and I don't believe that level of detail will be applied to self-driving vehicles, honestly. Government standards will be met and companies will achieve the best safety ratings they can because it just doesn't hurt your product to have that as a selling point(I purchased my last vehicle for my family almost solely on it's high safety ratings) but sooner or later a self-driving vehicle will cause a casualty or injury.
3. Control and change, it is a complete paradigm shift for transportation as a whole and it will take a while some people to want to adapt or change with the times, a lot of cars are driven for sentimental reasons, the self-driving car tends to pull away from the romantic image of a vehicle(road trips come to mind as an example).
4.Cost, Self-driving cars bring a whole new level of technology to the consumer, how will that be handled from a financial perspective? Car dealers aren't known to be consumer friendly. Will there be economy models available? What about the individuals who just can't afford a self-driving car and are fine with a non-self-driving vehicle, individuals who can't afford to purchase a new car at all?