edw, you raise a good point. Keep in mind that Apple has hundreds of development teams. Each one is different. I don't think I disclosed any secrets and I was very careful to not say anything that might embarrass Apple.
I think Guy Kawasaki's first book, The Macintosh Way, goes much further than my comments here – and he was even rehired back at Apple several years after his book was published.
My hope is two fold. 1. That someone reading this would pick up some good practices; and 2. That someone reading this would say, "I want to be a part of that!"
Apple's a great company to work at on so many different levels.
Yup, we turn off the online Apple store during the Steve Jobs keynote. Next time he speaks, check out store.apple.com and you'll see the yellow "We'll be back soon" sticky. If my memory serves, it's been like that for at least ten years.
Yup, I have to agree with you that I was very happy to work on a team with professional standards. While commonplace at high tech organizations, there are many non-tech organizations with their own tech teams which aren't aware of good practices since it's something that you have to experience, rather than be taught from a text book. That non-tech group is the one I'm really hoping to reach.
For some reason, binary trees look real cool when you learn about them from a textbook that people sometimes forget about hash tables. But, that might just be my personal experience from candidate interviews.
In my experience, large high tech organizations have many environments such as dev, build, QA, staging, demo, live, etc. The really good organization, with the smart engineers, have built this environments over the span of years and they've carefully thought through how each environment works.
Also, some organizations, especially ones that use cloud computing (EC2, Slicehost, etc) have the option of creating a staging environment for QA. testing, etc and, if everything checks out, the staging environment can be restarted as a live environment (there are ways to do this safely and securely) while leaving the previous live environment live. If, for some reason, you find problems with the new live environment, then you can simply shut it down and keep the old live environment humming along. Of course, this is easier said than done, but it'll keep the anxiety level down.
Another good idea is to have apps that know which environment they're in when they start up. So, if an app starts up in dev, it should only access the dev database. If it starts up in the live deployment environment, it should only access the live database. To enforce this security, the live database must only allow connections from clients in the live environment (IP address).
But, these "self aware" apps don't necessarily need (probably should not) have configuration files with the sensitive data used to access databases, credit card processors, etc. Rather, at app start up, these apps should know which environment they're in and then go over the network to fetch the sensitive data used for logging into databases, services, etc. (The apps should only store this sensitive data in memory.) The services, on the other end of that connection, should be checking the IP address of the app which is asking for the data to ensure that there isn't a mixup.
I have to second BrandomM's comments. Don't make a complete copy of your live database to use in dev. Sensitive data needs to be scrubbed, ideally by a bona fide DBA, before bringing it over. Best to avoid this if at all possible so you don't have to worry about overlooking something that was sensitive.
Also, I don't recall ever seeing any security issues with our environment or how we handled the code and data. There were some very smart engineers "minding the store."
You raise a great point that I wished I had clarified. I worked on the product side of the store, not, say order fulfillment, order integration, etc.
Multiple databases made up the live environment. The developers did not have access to the live environment. As a matter of fact, we had developed it to the point that, except in special circumstances, the engineers didn't even know what products were about to be announced until Steve Jobs announced them to the public.
The dev database was a copy of the products database – most everything in the products database was either public information or it was obsolete data for EOL Apple and third party products. I don't recall ever seeing any sensitive data in our production environment. Unlike, say Walmart or Amazon, we didn't need to keep track of inventory since we build the products (the only exception that comes to mind were the refurbed products since there was a limited amount of inventory).
Dev systems would process dummy credit card numbers, etc, in dev (but I didn't work, in that area). The applications were smart enough to figure out which environment they were in when started up which is a big help.