I completely agree that there is way too much regulation. But, the other part of the equation is people are still willing to pay the resulting price in the end. In the more "prime" communities like Palo Alto, Menlo Park, and Mountain View they still seem happy to pay the extra cost to get permits to tear down a 60 year ranch style home and then petition a city council to put a custom build in its place. Contractors advertise that they know the regulations and how to work with officials to get their permits through and charge a premium. The funny part is once construction is complete the new residence want to prevent new construction in order to "preserve the neighborhood" they moved into.
I live in the Bay Area. I have been able to make it work for me and love it here, though I do understand some of the reasons why others may not. Cost probably tops that list. But, the complaints about traffic should not be focused on California. In my experience most of the cities I have to travel to (Seattle, Phoenix, Atlanta) have much worse traffic issues. My colleagues in those cities spend 5-10 extra hours driving during the week compared to what my coworkers and I spend.