The anatomy of the perfect technical interview from a former Amazon VP(firstround.com)
firstround.com
The anatomy of the perfect technical interview from a former Amazon VP
http://firstround.com/article/The-anatomy-of-the-perfect-technical-interview-from-a-former-Amazon-VP
117 comments
I remember my own interview process with Amazon. I wasn't interviewing for an SDE position but a TPM role. I made it through the gauntlet and it all fell down due to Amazon's grossly unrealistic understanding of what the going rates for experienced PMs are.
I mostly had fun with it, but I was also frustrated.
The phone screens were the hardest in a purely technical sense, lots of algorithm questions that really didn't have anything to do with the job at all, I had to reach into my way-back machine to dig up some of the runtimes. But I made it through three of phone calls like that and had a bit of fun. After the phone calls and a chat with a senior person in the department, I was pretty sure I had a feel for where I was going to end up and what kind of projects I'd be managing. So I boned up on management theory and algorithms and a few other odds and ends.
Flying out to Amazon for the face-to-face, I realized I was wrong. Each face-to-face interview was wildly unfocused, I was interviewing in subjects the interviewer often didn't have any experience in and completely didn't reflect the nature of the phone interviews or the statements I had been given by the senior guy I spoke with. In one interview I'm designing Java APIs for movie tickets, in another I'm specifying contingency plans for overseas warehouse fulfillment issues. In another I was asked by two very junior people who frowned in confusion the entire time as I discussed complex employee and co-worker conflict resolution cases I've experienced over the last 18 years. In fact I was never asked a question about the first 15 years of my work experience, most of which would have been more relevant. But I was asked tons of questions about my side-startup, which was just me and my wife hacking around on the side and getting passive ad revenue and had no relevancy at all to the job.
The "lunch chat" was with somebody who couldn't have made small talk if he had a gun to his head. He just sat in silence and ate his soup the entire time while I uncomfortably ate one of the sloppiest turkey pesto sandwiches I've ever encountered.
I definitely didn't nail the face-to-face, but that's because I was playing trivial pursuit with the interviewers and the answers were arbitrary, instead of knuckling down on the job, the role, the department, the expectations of performance and the responsibilities of the position.
But I did something right because I was called in the next day to go over compensation and was blown away to find out the absolute max pay cap for the entire company was almost a 20% pay cut from what I'd been making in my previous job (a job that I was leaving because I was 10-15% under my peers and had been in a pay freeze for 3 years while the company worked towards profitability -- and yes it was from a metro area with a similar cost of living to Seattle).
Despite that, Amazon would have been good to work at, but even my absolute pay floor was 10% over. I even asked to make up the compensation difference with a mix of stocks or other methods, but eventually we had to agree that there was no way I could live the life I had become accustomed to with what Amazon was willing to pay.
I mostly had fun with it, but I was also frustrated.
The phone screens were the hardest in a purely technical sense, lots of algorithm questions that really didn't have anything to do with the job at all, I had to reach into my way-back machine to dig up some of the runtimes. But I made it through three of phone calls like that and had a bit of fun. After the phone calls and a chat with a senior person in the department, I was pretty sure I had a feel for where I was going to end up and what kind of projects I'd be managing. So I boned up on management theory and algorithms and a few other odds and ends.
Flying out to Amazon for the face-to-face, I realized I was wrong. Each face-to-face interview was wildly unfocused, I was interviewing in subjects the interviewer often didn't have any experience in and completely didn't reflect the nature of the phone interviews or the statements I had been given by the senior guy I spoke with. In one interview I'm designing Java APIs for movie tickets, in another I'm specifying contingency plans for overseas warehouse fulfillment issues. In another I was asked by two very junior people who frowned in confusion the entire time as I discussed complex employee and co-worker conflict resolution cases I've experienced over the last 18 years. In fact I was never asked a question about the first 15 years of my work experience, most of which would have been more relevant. But I was asked tons of questions about my side-startup, which was just me and my wife hacking around on the side and getting passive ad revenue and had no relevancy at all to the job.
The "lunch chat" was with somebody who couldn't have made small talk if he had a gun to his head. He just sat in silence and ate his soup the entire time while I uncomfortably ate one of the sloppiest turkey pesto sandwiches I've ever encountered.
I definitely didn't nail the face-to-face, but that's because I was playing trivial pursuit with the interviewers and the answers were arbitrary, instead of knuckling down on the job, the role, the department, the expectations of performance and the responsibilities of the position.
But I did something right because I was called in the next day to go over compensation and was blown away to find out the absolute max pay cap for the entire company was almost a 20% pay cut from what I'd been making in my previous job (a job that I was leaving because I was 10-15% under my peers and had been in a pay freeze for 3 years while the company worked towards profitability -- and yes it was from a metro area with a similar cost of living to Seattle).
Despite that, Amazon would have been good to work at, but even my absolute pay floor was 10% over. I even asked to make up the compensation difference with a mix of stocks or other methods, but eventually we had to agree that there was no way I could live the life I had become accustomed to with what Amazon was willing to pay.
The pay gap is probably because you did not pass the bar at the same "level" you were at in your old company, meaning Amazon was bringing you in at a demotion essentially.
[deleted]
> couldn't have made small talk if he had a gun to his head
To be fair, I think this describes most people.
To be fair, I think this describes most people.
It describes most technical people who are also introverted and spent most of their lives with minimal face-to-face social interaction.
Most people in general. A lot of people think they're much better at small talk than they are, and blame the other guy when an interaction doesn't go as smoothly as they hoped, or there's an awkward silence.
People who are excellent at small talk have to be very charismatic and seem genuinely interested in the conversation even when it's very boring. Very few people are capable of either of these things, and in my experience it's actually management & marketing types who are the worst at both.
People who are excellent at small talk have to be very charismatic and seem genuinely interested in the conversation even when it's very boring. Very few people are capable of either of these things, and in my experience it's actually management & marketing types who are the worst at both.
OT, but I just meant that one would be preoccupied with the gun to one's head.
My understanding is when they fly you out, you're pretty much a shoe-in and now they're letting their devs have a shot at trying to interview, while getting a feel how you work with others. i.e. are you sane or will you eat on of our disposable dev's face?
Its generally the opposite, the phone interviews are the sanity check.
"sanity check" in this context is a bit different than in software.
This was not my experience. I did not get an offer after being flown out and spending the day interviewing. I did not do well in the whiteboard coding, but I found the people I interacted with to generally be interested and smart.
There's one thing this article misses, and one thing I totally disagree on.
Hiring for prior experience - the author emphasizes this, and based on my experience this should be de-weighted. The issue is that as soon as you start to define an area of experience, your hiring pool shrinks dramatically. Consequently you are more likely to pick a less powerful engineer. Another, bigger downside here is that companies are terrible at figuring out what they really need. Consequently you've hired someone who can do A and B but one month in you realize that the thing you were missing is really C. A less powerful engineer will have a harder time transitioning over to doing C work, and you'll be worse off. Moreover they might be upset - they came in with the expectation of doing A & B, and they're sitting here doing something unrelated instead. Can't count the number of times I've seen this happen, to both the employee's and company's detriment.
The thing that this article misses is how to avoid hiring jerks. Terrible employees can be a net negative to a company. Jerks, particularly those in senior positions, are the thermonuclear version of terrible employees. They can demotivate and destroy the productivity of entire teams and cause your best engineers to rage-quit. The best interviewing tactic I've seen to weed out these people is watching how someone behaves when you "play dumb" in an interview. See if they're happy to explain something to you, or if they get impatient with your lack of knowledge. A related trick is challenging them on something that is obviously correct - see if they carefully prove why their answer is correct, or become antagonistic. Really smart people are good, but much more effective are those that can help bring everyone else up to their level.
Hiring for prior experience - the author emphasizes this, and based on my experience this should be de-weighted. The issue is that as soon as you start to define an area of experience, your hiring pool shrinks dramatically. Consequently you are more likely to pick a less powerful engineer. Another, bigger downside here is that companies are terrible at figuring out what they really need. Consequently you've hired someone who can do A and B but one month in you realize that the thing you were missing is really C. A less powerful engineer will have a harder time transitioning over to doing C work, and you'll be worse off. Moreover they might be upset - they came in with the expectation of doing A & B, and they're sitting here doing something unrelated instead. Can't count the number of times I've seen this happen, to both the employee's and company's detriment.
The thing that this article misses is how to avoid hiring jerks. Terrible employees can be a net negative to a company. Jerks, particularly those in senior positions, are the thermonuclear version of terrible employees. They can demotivate and destroy the productivity of entire teams and cause your best engineers to rage-quit. The best interviewing tactic I've seen to weed out these people is watching how someone behaves when you "play dumb" in an interview. See if they're happy to explain something to you, or if they get impatient with your lack of knowledge. A related trick is challenging them on something that is obviously correct - see if they carefully prove why their answer is correct, or become antagonistic. Really smart people are good, but much more effective are those that can help bring everyone else up to their level.
Raw smarts is one thing, and experience with a particular platform/problem space is another, and both are valuable on the job, and both should be weighted in hiring.
Sometimes it's appropriate to hire a brilliant person who's unfamiliar with your problem space, but not often. Sometimes you need an expert who already knows what they're doing and doesn't need to ramp up on the company's dime.
Sometimes it's appropriate to hire a brilliant person who's unfamiliar with your problem space, but not often. Sometimes you need an expert who already knows what they're doing and doesn't need to ramp up on the company's dime.
Great point about not hiring jerks, and those are some excellent ideas about how to avoid it. Having worked with, and currently working with, that kind of jerk I've seen how destructive they can be.
Being partly facetious here: I don't know how you actually propose not to hire jerks. The industry is full of jerks. Jerks are doing the interview much of the time. The industry wants people who claim to be the best, and that's hard to do while also selecting for niceness.
Personally, I have a bias against subjects who claim to be the best. Some of the best people I've worked with, would say things like, "Oh, I guess I'm ok at that" and then blow me away. It's the Dunning-Kruger effect [1] in action.
Because of that, I generally avoid people to self-evaluate versus others. I think it's ok to ask them to compare on purely internal things (e.g., "What languages are you strongest in"); people seem pretty good at that.
I think that approach makes it a lot easier to hire nice people; niceness and humility are correlated.
[1] http://en.wikipedia.org/wiki/Dunning–Kruger_effect
Because of that, I generally avoid people to self-evaluate versus others. I think it's ok to ask them to compare on purely internal things (e.g., "What languages are you strongest in"); people seem pretty good at that.
I think that approach makes it a lot easier to hire nice people; niceness and humility are correlated.
[1] http://en.wikipedia.org/wiki/Dunning–Kruger_effect
I like Lao Tzu's take on this:
'Those who know don't talk. Those who talk don't know.'.
'Those who know don't talk. Those who talk don't know.'.
The industry wants people who claim to be the best, and that's hard to do while also selecting for niceness.
Absolutely true. The reason I point this out is that the cost of a jerk goes up as the size of the team he/she is on increases. If you are a 3-man start-up, hiring a superstar coder who's sometimes a PITA can be worthwhile - have them work on some part of the project where they drive the whole thing. At 20 people, that person can become a huge liability - they will misdirect the team, demoralize other engineers, and eventually cause your best people to leave.
My examples illustrate how you might identify people with an uncontrollable mean streak during the candidate process, and hence filter them out. If you're building a scalable organization you must start with founders that aren't jerks and weed these people out of the candidate pool. If you start the company off with such a person, well, either live with it or find a new company.
Absolutely true. The reason I point this out is that the cost of a jerk goes up as the size of the team he/she is on increases. If you are a 3-man start-up, hiring a superstar coder who's sometimes a PITA can be worthwhile - have them work on some part of the project where they drive the whole thing. At 20 people, that person can become a huge liability - they will misdirect the team, demoralize other engineers, and eventually cause your best people to leave.
My examples illustrate how you might identify people with an uncontrollable mean streak during the candidate process, and hence filter them out. If you're building a scalable organization you must start with founders that aren't jerks and weed these people out of the candidate pool. If you start the company off with such a person, well, either live with it or find a new company.
> There is no room in the Amazon interview process to "try out" a candidate for a week or two in a real world setting to see how they perform.
When looking for work, I have no room for companies that want to do this: I already have a job and commitments. Also, I might not even be in the same state as the company I am talking to.
When looking for work, I have no room for companies that want to do this: I already have a job and commitments. Also, I might not even be in the same state as the company I am talking to.
This is such a popular and I think corrosive meme that I think it's a service to the community when people pipe up and point out the obvious, which is that many/most people can't even afford to do try-outs, let alone be happy about the idea. Thanks!
How is trying out a practical for anything other than unemployed new grads? I see this constantly suggested but I don't know a single person who would even consider quitting or taking vacation time to go work for a prospective company.
Although this would work for people who are unemployed. I took a 3 months contract position, and I was hired full time after 1 year. It's been 2.5 years and I'm still at the company.
The only way I'd consider doing this is taking vacation if I'm being paid a rate that is significantly higher than my current rate.
Otherwise, just ask for a coded solution to some specific problem or user story and I'll work on it after hours and send it in.
Otherwise, just ask for a coded solution to some specific problem or user story and I'll work on it after hours and send it in.
Depends a lot on the person. If somebody has done consulting or contract work, it's a pretty natural thing. I've done it both as an employee and an employer.
I barely even have time for the 3-6 technical interviews most companies want to do, let alone multiple days/WEEKS of "try out" period.
I used to live about 8 hours drive from where I'm currently working (rural australia to Sydney). When they found out where I was, they arranged the normal 4 interviews to be arranged back to back rather than spread out over a few days.
Just the fact they were willing to do that meant a lot for me (because no one else had ever bothered how inconvenient it was for me to do interviews).
Still here 3 years later and the attitude shown during the interview process extended right through the company.
Just the fact they were willing to do that meant a lot for me (because no one else had ever bothered how inconvenient it was for me to do interviews).
Still here 3 years later and the attitude shown during the interview process extended right through the company.
I think the comment is just being misunderstood, but Amazon (at least AWS), doesn't do any try before you buy type stuff. It is usually an hour long initial phone screen, than an all day long 7 hour interview with multiple people over the course of a single day in Seattle.
Although there are definitely logistical problems with this approach, I've had a couple of jobs now where I wish I would have been able to "try it out" for a couple of weeks. Having done so, I never would have accepted the offer (saving me a lot of frustration and a couple of large relocations).
> I think Amazon's interview process is fatally flawed.
The topic of interviewing comes up often here, and this sentiment is overwhelmingly the top sentiment for every single company's interview process. Amazon, Google, Apple, Facebook, Microsoft, etc., have all been pilloried here about the interview process.
I suspect the real problem is the implication that this is even a solvable problem.
The topic of interviewing comes up often here, and this sentiment is overwhelmingly the top sentiment for every single company's interview process. Amazon, Google, Apple, Facebook, Microsoft, etc., have all been pilloried here about the interview process.
I suspect the real problem is the implication that this is even a solvable problem.
I agree. I don't think it's a solvable problem. It's like people who champion the removal of SAT/ACT from college admissions.
Okay, so statistics demonstrate the SAT only correlates with the first year of college...great, what do you have to replace it with?
We work with what we have, and take the benefits it gives us.
Systems can always be gamed, and that's an unfortunate (fortunate for some, I guess) reality.
Okay, so statistics demonstrate the SAT only correlates with the first year of college...great, what do you have to replace it with?
We work with what we have, and take the benefits it gives us.
Systems can always be gamed, and that's an unfortunate (fortunate for some, I guess) reality.
It's a maximization problem. The goal is to have the best possible ROC curve for your hiring process. Maximize true positives while minimizing false positives.
My personal threshold attempts to minimize false positives even to the extent that I will pass on people that would probably have done a good job. I tune it this way because I think that bad hires are one of the worst things that can happen to a team. But, each individual has to calibrate this for themselves.
I'm not looking to solve the problem, I'm looking to make the best possible decision based on the information I have available.
My personal threshold attempts to minimize false positives even to the extent that I will pass on people that would probably have done a good job. I tune it this way because I think that bad hires are one of the worst things that can happen to a team. But, each individual has to calibrate this for themselves.
I'm not looking to solve the problem, I'm looking to make the best possible decision based on the information I have available.
That sounds accurate, but
1. You need large samples of hires because you must know what features actually correspond with good performance
2. You need to have a near perfect understanding of statistics
Once Google and friends collected a large enough sample they concluded their puzzles are not a good indicator. Similarly, people tend to have bias towards certain skills without any statistical evidence that they are correlated with performance. And statistics is all about removing the bias, which has proven time and time again is very hard. Otherwise you would be just gambling on instinct, which at best is just a sanity check.
1. You need large samples of hires because you must know what features actually correspond with good performance
2. You need to have a near perfect understanding of statistics
Once Google and friends collected a large enough sample they concluded their puzzles are not a good indicator. Similarly, people tend to have bias towards certain skills without any statistical evidence that they are correlated with performance. And statistics is all about removing the bias, which has proven time and time again is very hard. Otherwise you would be just gambling on instinct, which at best is just a sanity check.
I agree strongly. But part of the problem with this solution is that you might be caught with a stream of merely adequate people you won't hire because you're afraid of false positives. I also think it's sometimes okay to hire merely adequate people, if you know ahead of time you can quarantine their usefulness to one area they seem stellar at.
3. Amazon considers all SDEs to be fungible but internal politics determines which group gets to interview a new candidate. Have an advanced degree and strong interest in machine learning? Too bad, US Retail needs someone to work on front-end development and they'll ask you lots of questions about javascript.
Yeah, this is pretty painful. I've phone interviewed with Amazon twice. At the time my resume was pure front-end developer with just a smattering of server-side experience. From the Amazon recruiter everything was smiles and 'perfect'.
The interview was for a server-side & database developer position with warehouse software. WTF?! Did you look at my resume for even twenty seconds? I got database design and management questions mostly.
That interview was brutal. What a waste of time for both parties.
Yeah, this is pretty painful. I've phone interviewed with Amazon twice. At the time my resume was pure front-end developer with just a smattering of server-side experience. From the Amazon recruiter everything was smiles and 'perfect'.
The interview was for a server-side & database developer position with warehouse software. WTF?! Did you look at my resume for even twenty seconds? I got database design and management questions mostly.
That interview was brutal. What a waste of time for both parties.
> There is no room in the Amazon interview process to "try out" a candidate for a week or two in a real world setting
Are you seriously proposing that any serious applicant would be willing to take a week off from their current job just to interview for a possible job at Amazon?
Are you seriously proposing that any serious applicant would be willing to take a week off from their current job just to interview for a possible job at Amazon?
Re 2.: I have taken some code challenges in the past and I find the Amazon open-ended questions the best. Sure, some people can (and possibly like to?) memorize different data structures and algos, but broad thinking suits me best and companies like Google and Facebook don't seem to be doing it enough.
Basing it on the interview questions posted online.
Basing it on the interview questions posted online.
If you have gaps in your algo knowledge check out this book (http://www.algorist.com/), it's a dense read but comprehensive. Why be a broad thinker when you can be an everything thinker?
Thanks. Even just the website gives a nice outline of some of the different problems and how to solve them.
Gaps in my algo knowledge are there on purpose since to me they are a solved problem and easily found online.
Gaps in my algo knowledge are there on purpose since to me they are a solved problem and easily found online.
I was just hired at Amazon as an SDE and found the interview process to be fairly challenging and technical (but what do I know).
I appreciated the fact that a lot of the interview focused on core data structures / algorithms and not languages / frameworks of the moment kind of thing. It caused me to go back through my university books and study a lot of the core concepts that I enjoyed learning and have gotten away from by doing more menial programming tasks (not to say that I won't be don't those sorts of things at the job).
Also, if they said that they wanted to try me out for a couple weeks I would have refused the offer. I already had a job that I was happy (and well paid) at, so why should I leave or take vacation to move across the country and be tried out at another company?
I appreciated the fact that a lot of the interview focused on core data structures / algorithms and not languages / frameworks of the moment kind of thing. It caused me to go back through my university books and study a lot of the core concepts that I enjoyed learning and have gotten away from by doing more menial programming tasks (not to say that I won't be don't those sorts of things at the job).
Also, if they said that they wanted to try me out for a couple weeks I would have refused the offer. I already had a job that I was happy (and well paid) at, so why should I leave or take vacation to move across the country and be tried out at another company?
Regarding point 3, I've seen that given out as an option before in talks about interviews. I only see it as a solution that works if the person is actually hired.
If not they waste 2 weeks of their time and probably other SDE time. It leaves no room for people who apply while still working at another job. I had to miss 2 days of work for my interview, and that was about the biggest risk I could afford. Even if you say you will pay me for the 2 weeks, I still wouldn't have a job to come back to. It's probably still a poor job, I was just starting to get into my first task towards the end of 2 weeks. I spend most of the time just getting everything setup (granted we ran into issues that had to be resolved with another team, but still).
If not they waste 2 weeks of their time and probably other SDE time. It leaves no room for people who apply while still working at another job. I had to miss 2 days of work for my interview, and that was about the biggest risk I could afford. Even if you say you will pay me for the 2 weeks, I still wouldn't have a job to come back to. It's probably still a poor job, I was just starting to get into my first task towards the end of 2 weeks. I spend most of the time just getting everything setup (granted we ran into issues that had to be resolved with another team, but still).
I'm always amazed by how confidently people can propose a hiring "rule" or even guideline.
The article pays lip service to the fact "There's no way to A/B test hiring decisions" but then goes on to describe a handful of rules to "build an effective organization, regardless of size or resources." The thing is, there's no evidence that these are effective and definitely none to suggest they're generalizable to organizations of every size. There's an appeal to the author's authority as an Amazon VP (admittedly a hell of a credential), but that's not evidence of effectiveness.
The fact is, hiring is tough to learn. It's made especially difficult because your feedback comes at least 6 months after you make the decision. I've had the good fortune to be in an organization which grew from ~10 employees this time last year to ~80 now, so I've gotten to see how a few dozen hires panned out in a short period of time. I learned a lot through that. Even so, as a professional data geek, I couldn't say with a straight face that I "know" anything about hiring. I do my best day in and day out, but saying you've got hiring all figured out is the professional equivalent of knowing "What (wo)men want." It's a sign of confidence, but not necessarily knowledge.
Don't get me wrong, in the absence of quality data, I'll take the wise words of someone who has been there and done that. But I'd much rather some empirical proof or at least some "YMMV" humility from those who purport to know.
The article pays lip service to the fact "There's no way to A/B test hiring decisions" but then goes on to describe a handful of rules to "build an effective organization, regardless of size or resources." The thing is, there's no evidence that these are effective and definitely none to suggest they're generalizable to organizations of every size. There's an appeal to the author's authority as an Amazon VP (admittedly a hell of a credential), but that's not evidence of effectiveness.
The fact is, hiring is tough to learn. It's made especially difficult because your feedback comes at least 6 months after you make the decision. I've had the good fortune to be in an organization which grew from ~10 employees this time last year to ~80 now, so I've gotten to see how a few dozen hires panned out in a short period of time. I learned a lot through that. Even so, as a professional data geek, I couldn't say with a straight face that I "know" anything about hiring. I do my best day in and day out, but saying you've got hiring all figured out is the professional equivalent of knowing "What (wo)men want." It's a sign of confidence, but not necessarily knowledge.
Don't get me wrong, in the absence of quality data, I'll take the wise words of someone who has been there and done that. But I'd much rather some empirical proof or at least some "YMMV" humility from those who purport to know.
This topic has started taking on religious tones in the past year or so. It seems to be a long stream of anecdotes and opinion pieces presented with appeals to authority either of the author or someone the author is connected to.
I appreciate the notion of having the whole team prepared for the interview process. I think this is a best practice. But, when I interview, I am looking for completely different things than the author.
I really don't care about what you did at your last company or if you raised this by 50% or that by 30%. Rather, I want three things in a candidate: smart, talented and hungry. If I get someone who is smart, talented and hungry and I drop them into a situation where there is unlimited upside, my job then becomes to guide them to greatness.
I also strongly disagree with the notion of not asking questions you do not know the answer to. Anyone can read an obscure section of the C++ language spec or a chapter on some data structure 5 minutes before an interview and make themselves feel superior to the candidate. What does this prove? Nothing. I often ask questions I don't know the answer to in order to determine whether or not the candidate can teach me something. I love having people on my team that have the ability to teach others.
The author's point is very well taken -- interviewing is an important process and it should be conducted in a way that results in the best possible outcome. But, he's looking at the situation far too rigidly for my taste.
I really don't care about what you did at your last company or if you raised this by 50% or that by 30%. Rather, I want three things in a candidate: smart, talented and hungry. If I get someone who is smart, talented and hungry and I drop them into a situation where there is unlimited upside, my job then becomes to guide them to greatness.
I also strongly disagree with the notion of not asking questions you do not know the answer to. Anyone can read an obscure section of the C++ language spec or a chapter on some data structure 5 minutes before an interview and make themselves feel superior to the candidate. What does this prove? Nothing. I often ask questions I don't know the answer to in order to determine whether or not the candidate can teach me something. I love having people on my team that have the ability to teach others.
The author's point is very well taken -- interviewing is an important process and it should be conducted in a way that results in the best possible outcome. But, he's looking at the situation far too rigidly for my taste.
I disagree with the hungry part (if you are referring to someone that needs the money). Being "hungry" just means that they will be stuck with a company but will not love what they do.
Less about needing money and more about being ambitious. Hungry for success I should say.
Whether hungry for money or hungry for success, you're then missing hiring candidates who aren't "hungry" because they have a proven track record at having done very well financially from a series of successes.
At this point, they don't need money or validation, they just figure they can help you solve things and are talking to you because they enjoy solving new things.
Hungry is overrated when what you need is focus on solving the business' problems. This is one reason VC like to see founders take "shelter money" off the table early.
At this point, they don't need money or validation, they just figure they can help you solve things and are talking to you because they enjoy solving new things.
Hungry is overrated when what you need is focus on solving the business' problems. This is one reason VC like to see founders take "shelter money" off the table early.
Someone who has sufficient self-made wealth to retire, or nearly retire, but who is still looking to work for you as a developer I would certainly describe as "hungry" in this context. But I'm mostly interpreting that term to mean "passionate" about the intellectual and creative aspects of the job.
Point taken. Sometimes I walk out of an interview and say to myself that the candidate did well, but just did not convince me that he/she really wanted the job. It may just be my personal perspective, but candidates that really stick the landing with me are ones that have that "let me at 'em coach!" attitude.
> [he] believes every phase of the process needs to be meticulously designed to drill deep into skill sets, ... and leadership potential
Ugh. I hate this. For a technical interview, why is leadership potential a priority? Speaking for myself, I've done the leadership thing and don't care for it. People tell me I'm good at it, but I still prefer dealing with algorithms over humans, and I actively decline offers for "promotion". (I already have my very own batch of little humans of to deal with, thank you very much.)
Unfortunately, no matter where you go, there's always the deep-seated cultural mindset that manager > worker. In terms of the corporate efficiency calculation of effectiveness/hour, I suppose that's still true. But for me, personally, that's not the equation that I care to maximize.
Ugh. I hate this. For a technical interview, why is leadership potential a priority? Speaking for myself, I've done the leadership thing and don't care for it. People tell me I'm good at it, but I still prefer dealing with algorithms over humans, and I actively decline offers for "promotion". (I already have my very own batch of little humans of to deal with, thank you very much.)
Unfortunately, no matter where you go, there's always the deep-seated cultural mindset that manager > worker. In terms of the corporate efficiency calculation of effectiveness/hour, I suppose that's still true. But for me, personally, that's not the equation that I care to maximize.
I agree re: the cultural mindset. And I definitely get preferring algorithms over humans. But I don't think leadership necessarily has to = management. Leadership can simply be about being persuasive, degree to which you deal with hurdles/stupidity/whatever without killing somone, able to facilitate a contentious discussion/meeting etc. - those soft skills that make you a good teammate, defacto team lead if/when necessary or whatever. Like you, I've been a manager and didn't like it (although I did like the fact that my input carried more weight than it would have otherwise). But I'm sure that leadership potential serves you well, even in non-management positions.
I do agree that that form of leadership is necessary. I was inferring a more literal interpretation of leadership, though, considering the associated word, "potential".
> For a technical interview, why is leadership potential a priority?
Probably because in many modern ways of organizing knowledge work, distributing leadership roles is an important part of the way organizations operate.
> Unfortunately, no matter where you go, there's always the deep-seated cultural mindset that manager > worker.
The places where that is least true are also the places where evaluating leadership potential in technical workers is the most relevant to immediate job performance.
Probably because in many modern ways of organizing knowledge work, distributing leadership roles is an important part of the way organizations operate.
> Unfortunately, no matter where you go, there's always the deep-seated cultural mindset that manager > worker.
The places where that is least true are also the places where evaluating leadership potential in technical workers is the most relevant to immediate job performance.
Probably because in many modern ways of organizing knowledge work, distributing leadership roles is an important part of the way organizations operate
Does it work, by which does it produce the desired results or benefits? It seems somewhat likely that this would have a side-effect of pushing employees to asserting a dominating personality.
Does it work, by which does it produce the desired results or benefits? It seems somewhat likely that this would have a side-effect of pushing employees to asserting a dominating personality.
but it is extremely hard (perhaps impossible) to accurately determine leadership potential in an interview
Amazon has a set of leadership principles that apply to everyone at the company, whether you are technical or not.
http://www.amazon.com/Values-Careers-Homepage/b?ie=UTF8&node...
It's being a leader in what you do, which isn't necessarily the same as leading a group of other people.
http://www.amazon.com/Values-Careers-Homepage/b?ie=UTF8&node...
It's being a leader in what you do, which isn't necessarily the same as leading a group of other people.
"No matter how many times they say it, most still make decisions based on gut feel, basic credentials, GPAs, ivy league educations, flashy company names - even SAT scores. Roseman objects."
What's a Roseman object? Is it some psychological classification of the means by which we just things? Of course, I mis-read it. Shame, that could have been an interesting topic.
What's a Roseman object? Is it some psychological classification of the means by which we just things? Of course, I mis-read it. Shame, that could have been an interesting topic.
I think there should be something in science called the 'reindeer effect.' I don't know what it would be, but I think it'd be good to hear someone say, 'Gentlemen, what we have here is a terrifying example of the reindeer effect.'
- Jack Handey
>SAT scores
Wait...what? Is this a thing? Are people actually putting this on their resumes?
I had the highest SAT in my county (still just a 1360/1600), but I don't go around putting 2002 "STAR student"[1] on my resume.
[1]Local award for highest SAT score. I think each school in Georgia awards this.
Wait...what? Is this a thing? Are people actually putting this on their resumes?
I had the highest SAT in my county (still just a 1360/1600), but I don't go around putting 2002 "STAR student"[1] on my resume.
[1]Local award for highest SAT score. I think each school in Georgia awards this.
Most undergrads right out of college will have a standardized test score on their resume. Usually it's the SAT or the GRE.
"The anatomy of the perfect technical interview from a former Amazon VP"
And more recently, a former Zynga VP.
Surprising that such an expert at evaluating people chose to work for Zynga, of all places.
And more recently, a former Zynga VP.
Surprising that such an expert at evaluating people chose to work for Zynga, of all places.
Money is a hell of a drug.
That's how you get the best talent. There is no secret question you can ask to find great talent.
I've interviewed persons that did very very well during the interview process, but failed miserably on a day-to-day basis.
I've interviewed persons that did very very well during the interview process, but failed miserably on a day-to-day basis.
So money is the answer? Have you never observed a well paid developer who "failed miserably on a day to day basis"?
Either that, or they really wanted a challenge.
There seems to be a lot of criticism of the suggestions here, but I think they are good ones. It seems to pretty much boil down to:
(1) Spend a lot of time discussing accomplishments at their current / previous employers. If you drill down on particular projects they should be able to explain, in great detail, exactly what their contribution was and how they worked with the rest of the team to get that done.
(2) Pick a technical question that has a lot of depth so that you avoid false positives because people have just looked up the answer on the internet. I would also add that in my view the best questions have no tricks and require no "aha!" moments. It should be something the candidate can be reasoned through step by step so you can see their thought process. When they come up with an initial solution, then you can dive deeper by expanding the problem or adding additional constraints.
(1) Spend a lot of time discussing accomplishments at their current / previous employers. If you drill down on particular projects they should be able to explain, in great detail, exactly what their contribution was and how they worked with the rest of the team to get that done.
(2) Pick a technical question that has a lot of depth so that you avoid false positives because people have just looked up the answer on the internet. I would also add that in my view the best questions have no tricks and require no "aha!" moments. It should be something the candidate can be reasoned through step by step so you can see their thought process. When they come up with an initial solution, then you can dive deeper by expanding the problem or adding additional constraints.
I'm also disappointed that so many comments above do not engage with the information in the article. I, too, found this to be a thoughtful and helpful piece. Along the parts I liked --
Using "calibrated" questions that you have used before and know how far people get with.
Focusing on a specific accomplishment or area of strength of the candidate to probe. (As opposed to choosing something from the resume that, say, the interviewer was most comfortable with, or that the interviewer found interesting.)
Paying attention to preparing the interviewers, so they know why the candidate is being considered, they know why they are on the interview list, and they are experienced enough with interviewing to give a useful opinion.
Using "calibrated" questions that you have used before and know how far people get with.
Focusing on a specific accomplishment or area of strength of the candidate to probe. (As opposed to choosing something from the resume that, say, the interviewer was most comfortable with, or that the interviewer found interesting.)
Paying attention to preparing the interviewers, so they know why the candidate is being considered, they know why they are on the interview list, and they are experienced enough with interviewing to give a useful opinion.
The one element no one ever accounts for is the interviewers. You're never going to have a perfect technical interview because you're an imperfect human. You're incapable of judging whether someone is a perfect cultural fit, and someone who doesn't fit the culture might help the company more than someone who does.
That isn't even getting into the the conflict between whats good for the company and whats good for you.
That isn't even getting into the the conflict between whats good for the company and whats good for you.
When Google pushed out their interview-results data this year, one of the big findings was that there is NO SUCH THING as a good interviewer.
To quote the NY Times interview with Laszlo Black of Google - "Years ago, we did a study to determine whether anyone at Google is particularly good at hiring. We looked at tens of thousands of interviews, and everyone who had done the interviews and what they scored the candidate, and how that person ultimately performed in their job. We found zero relationship. It’s a complete random mess, except for one guy who was highly predictive because he only interviewed people for a very specialized area, where he happened to be the world’s leading expert."
That being said, there are a lot of very disorganized interviewers out there, especially among seed stage companies. I've been amazed about how many people grumble about the "talent shortage", yet regularly miss interviews with "hot" candidates because of meetings, forget them altogether, or just lack a sense of urgency & speed when it comes to running an effective hiring process.
A large part of having a hiring culture is about decisiveness. Loosing a fantastic Engineer because you spent 4 weeks making up your mind, or you didn't want to pay an extra $10K/year is just silly when it means that you won't hit your product roadmap goals.
To quote the NY Times interview with Laszlo Black of Google - "Years ago, we did a study to determine whether anyone at Google is particularly good at hiring. We looked at tens of thousands of interviews, and everyone who had done the interviews and what they scored the candidate, and how that person ultimately performed in their job. We found zero relationship. It’s a complete random mess, except for one guy who was highly predictive because he only interviewed people for a very specialized area, where he happened to be the world’s leading expert."
That being said, there are a lot of very disorganized interviewers out there, especially among seed stage companies. I've been amazed about how many people grumble about the "talent shortage", yet regularly miss interviews with "hot" candidates because of meetings, forget them altogether, or just lack a sense of urgency & speed when it comes to running an effective hiring process.
A large part of having a hiring culture is about decisiveness. Loosing a fantastic Engineer because you spent 4 weeks making up your mind, or you didn't want to pay an extra $10K/year is just silly when it means that you won't hit your product roadmap goals.
I think that is perhaps telling. Google doesn't interview you on your experience, but whether you can can solve network/graph algorithms taken from Knuth exercises. I would expect a random mess from that. It sounds like this person actually drilled down into what the candidate actually knew.
I think it is far more useful to ask somebody about what they actually did. Are they excited and engaged by it, did they come up with novel thoughts about it, do they understand the CS literature in relation to it, and so on. Program a hash table on the whiteboard when you have been just fine using the built in one in your favorite library or language? I'm not sure what that predicts about my on job performance. If I had to do that I'd reread the literature and then whip one up, easy peasy. I can state that with confidence because I read the literature on other topics and do that every day. Have me explain the new numerical filtering technique I just implemented and I'll give you good interview. Ask me to implement a smart pointer? Not so much.
I think it is far more useful to ask somebody about what they actually did. Are they excited and engaged by it, did they come up with novel thoughts about it, do they understand the CS literature in relation to it, and so on. Program a hash table on the whiteboard when you have been just fine using the built in one in your favorite library or language? I'm not sure what that predicts about my on job performance. If I had to do that I'd reread the literature and then whip one up, easy peasy. I can state that with confidence because I read the literature on other topics and do that every day. Have me explain the new numerical filtering technique I just implemented and I'll give you good interview. Ask me to implement a smart pointer? Not so much.
Google doesn't interview you on your experience, but whether you can can solve network/graph algorithms taken from Knuth exercises.
I believe this is a backdoor method to requiring a college degree for positions that do not require one within labor law.
I believe this is a backdoor method to requiring a college degree for positions that do not require one within labor law.
Is there any labor law that would require you to use a back door method?
This doesn't make sense to me on first read. I'm unaware of a labor law that requires you to interview people without college degrees. Care to elaborate?
I may have worded it wrong. A large swath of programming jobs do not require a college degree, and in those cases it's illegal to require one. This is why job ads will say, "Desired: College degree or equivalent experience." They don't describe it as a requirement for a reason.
[deleted]
It's illegal to require one? Citation?
HN's linker requires you to manually add the period at the end:
http://en.wikipedia.org/wiki/Griggs_v._Duke_Power_Co.
http://en.wikipedia.org/wiki/Griggs_v._Duke_Power_Co.
If you design your interviews to focus on knowledge and abilities taught in college, you tend to cut out people without college degrees who don't have the personal grit to go teach themselves those subjects. Since most people don't care for academic learning, this effectively requires a college degree in general.
That said, the big G does hire a number of people without degrees, I hear.
That said, the big G does hire a number of people without degrees, I hear.
I disagree with "In most cases, the “best and the brightest” already have jobs, so you’re really just on the lookout for the best available.". Just because one is employed does not mean one is taken. If you want the best you have to work at it. Appeal to their desires and they'll come to you. Looking for the "best available" is no better than settling. In my opinion you'll end up doing a disservice to both your company and the new hire.
If you have a situation, which I believe we do now, where the number of engineers who fit the "best and brightest" category as defined by the companies looking is less than the number required by these companies, somebody is going to have to either "settle" or work with a hole. It will also make some of these engineers overpaid. I don't see how this benefits your company. Yes, in this system some companies will be "winners" in the sense that they get an awesome team for a good price, but most will lose. I would not be surprised if the high rate of failure in the startup community is related to this.
For the record, I do not see hiring the "best available" as settling unless they do not meet some minimum requirements of the business. The purpose of hiring people is to help execute a product, service, or whatever else your business is trying to make money off of. Anyone who can increase your chances of succeeding or increase your level of success is a good hire. Yes, that includes looking at intangibles like team and culture fit, work ethic, etc. But this person does not need to be the "best" to provide this boost to your business. In fact, it is effectively impossible to hire the best, because there will always be someone better.
For the record, I do not see hiring the "best available" as settling unless they do not meet some minimum requirements of the business. The purpose of hiring people is to help execute a product, service, or whatever else your business is trying to make money off of. Anyone who can increase your chances of succeeding or increase your level of success is a good hire. Yes, that includes looking at intangibles like team and culture fit, work ethic, etc. But this person does not need to be the "best" to provide this boost to your business. In fact, it is effectively impossible to hire the best, because there will always be someone better.
"over paid" what is this concept that you speak off - there is no big guy with a big white beard sitting on a cloud who's going to reach down and say "you yes you with a third in media studies from Luton university should be paid X and those cheeky blue collar train drivers or miners who have scarce skills should be paid less than you".
The PHB might think they are over paid but like it or not this is Laissez-faire capitalism pay is determined by scarcity and bargaining power.
ps for those outside the UK Luton is the unfortunate University that props up the bottom of the university league table - though to be fair they do rank well for Nursing degrees.
The PHB might think they are over paid but like it or not this is Laissez-faire capitalism pay is determined by scarcity and bargaining power.
ps for those outside the UK Luton is the unfortunate University that props up the bottom of the university league table - though to be fair they do rank well for Nursing degrees.
Paid more than one standard deviation higher than average for that level of competency, scarcity, and bargaining power.
If you could have hired someone equally competent for less money, why didn't you?
If you couldn't find such a person, the person you hired is hardly overpaid.
If you couldn't find such a person, the person you hired is hardly overpaid.
Quite thats the point I was making there is no God given "correct wage" just the market unless saraid216 thinks we should move to a fully communist model where the state says what job you get and how much you get paid?
No, no. The fully communist model is where you have listings of every single worker in existence and choose your hires based on that so that you have an accurate estimate of scarcity.
What are the levels of competency, scarcity, and bargaining power? How do you place a particular candidate in her appropriate levels?
Lets be honest Amazon is an online shop the best and brightest are either going to be working at tier 1 RnD or on wall street or the city.
Re Wall Street: It's unfortunate that some of our brightest are tempted into playing multi-million dollar poker, rather than focusing their energy on producing any significant long-term value.
I believe software engineering wages will continue to rise over time, and will eventually reflect the true value the industry produces.
I believe software engineering wages will continue to rise over time, and will eventually reflect the true value the industry produces.
I have no shame in admitting that I would rather use high-level mathematics and programming to play "multi-million dollar poker" on Wall Street than join half of the startups and big tech companies and have a boring job churning out insignificant apps at a desk.
In fact, the way you phrased it, being a quant sounds really fun :) you don't want to play multi-million dollar poker?
I mean, sure, meaning and existentialism and all that, but I bet it's at least somewhat exciting.
In fact, the way you phrased it, being a quant sounds really fun :) you don't want to play multi-million dollar poker?
I mean, sure, meaning and existentialism and all that, but I bet it's at least somewhat exciting.
That's the fundamental problem. The function of the finance system in a healthy economy is to ensure efficient allocation of capital. Playing poker has a neutral to negative effect on the system.
Well you're the one who used the word poker, so I guess it depends on your definition here...but the financial system somewhat revolves around poker.
Valuations of companies are only important to investors because of the inherent gambling. If you purchase capital at a single valuation that never rises or falls, what's the point? Even an index fund pursues gradual increases over time.
...Or am I building a strawman here? Is this not where you were going?
Valuations of companies are only important to investors because of the inherent gambling. If you purchase capital at a single valuation that never rises or falls, what's the point? Even an index fund pursues gradual increases over time.
...Or am I building a strawman here? Is this not where you were going?
Which is why I work for a large bank, in toxic assets resolution, where the stakes are still in the billions of dollars.
I think I'd rather use my tech skills to play multi-million dollar poker on Wall Street then create yet another beautiful, responsive, flexible, tweet-able, location-based, modular MVC library API component framework backbone architecture.js.
"It's beautiful! It's disruptive! It's agile and lean!"
"What does it do?"
"It revolutionizes the way you share pictures of your cats!"
...Yeah. Playing multi-million dollar poker on Wall Street sounds a lot sexier than the shiny startup veneer put on another photo sharing app.
"What does it do?"
"It revolutionizes the way you share pictures of your cats!"
...Yeah. Playing multi-million dollar poker on Wall Street sounds a lot sexier than the shiny startup veneer put on another photo sharing app.
Experience from other branches of engineering compared to other equivalent professions over the last 100 years suggests otherwise.
Let's be honest, Amazon has a massive infrastructure supporting a very large number of very diverse customers, tonnes of products and projects, and lots of active development on them.
Have you heard of a little thing called Amazon Web Services?
down voted eh the truth hurts doesn't it there is a world of difference between working at JPL MIT etal and some me to start up using the latest shiny new thing copying some existing SM type solution of which there are dozens already.
I didn't downvote you, but calling Amazon a startup is pretty ridiculous -- and your spelling/grammar isn't lending your argument any favors. (Though, ironically, it does fit into PG's description of "a company optimized for growth.")
I think it should have been obvious from the context that I was talking about a substantial number of the HN commentators who are working at me to start ups (nothing wrong with that but its not world changing rocket science)
Soory if you have problems with reading as a literate dyslexic I had no problem with reading writing not so much :)
Soory if you have problems with reading as a literate dyslexic I had no problem with reading writing not so much :)
[deleted](1)
cough http://aws.amazon.com/solutions/case-studies/nasa-jpl/ cough
Sorry, caught a nasty flu..
Sorry, caught a nasty flu..
Not related to the content, but why do sites like this disable pinchzoom? What value could it possibly add?
Yesterday I needed to look at source code on GitHub using my iPhone. Horizontal scrolling AND pinch-to-zoom were disabled, making it impossible to read lines of code wider than my phone screen.
Github.
Why is this so hard?
Github.
Why is this so hard?
For the same reason other sites use floats for sidebars: they hate you and want to become irrelevant.
You pass the interview process and then end up doing shit that monkey can be trained to do.
one of the most important things - something that is difficult to suss out in an interview - is how well you (or your team) would work with a person. (i think this is perhaps the most important quality - definitely more important than being able to solve brain teasers.)
i've wondered lately if it'd be better, during an interview, to choose a problem that both you and the candidate don't know...and try to solve it together...
i've wondered lately if it'd be better, during an interview, to choose a problem that both you and the candidate don't know...and try to solve it together...
Moving away from "the best available" towards "not the worst possible" (who could potentially destroy our company) would be a good idea.
What's notably lacking from this perfect interview is mention of referrals. I've had the best of luck with referrals as the managers I've worked with have never felt the need to "burn" me and could give an accurate assessment of the roles and responsibilities I have in my current job.
without any data and careful analysis of that data, his opinion on how to hire is worthless.
it's somewhat disturbing that he likes to use the STAR framework, which normally leads candidates into artificial responses. is there any reason to believe that this framework leads to better results?
it's somewhat disturbing that he likes to use the STAR framework, which normally leads candidates into artificial responses. is there any reason to believe that this framework leads to better results?
Here were the most frustrating aspects of interviewing for me:
1. Bar raisers don't ask questions that they themselves don't know the answer to. I believe this is what Roseman is referring to (as a desirable?!? attribute) when he says "One of the things that really pisses me off is people asking questions that they don’t even have a good handle on themselves". What happens in practice is that you miss hiring people who may not be able to answer a question to which every interviewer already knows the answer but whom might be uniquely qualified in some other area.
2. Interviewers still ask candidates to code answers to theoretical questions on the whiteboard. There is no room in the Amazon interview process to "try out" a candidate for a week or two in a real world setting to see how they perform.
3. Amazon considers all SDEs to be fungible but internal politics determines which group gets to interview a new candidate. Have an advanced degree and strong interest in machine learning? Too bad, US Retail needs someone to work on front-end development and they'll ask you lots of questions about javascript.
4. This isn't really part of the Amazon interview process but Amazon's job descriptions are (for the most part) written so generically and absent of personality that I think they miss out on lots of great candidates for whom startups simply seem more fun & interesting.