Lessons from a year’s worth of hiring data (2013)(blog.alinelerner.com)
blog.alinelerner.com
Lessons from a year’s worth of hiring data (2013)
http://blog.alinelerner.com/lessons-from-a-years-worth-of-hiring-data/
43 comments
In fact, this is the 4th time (See here: https://hn.algolia.com/?query=http:%2F%2Fblog.alinelerner.co...)
It's a well researched article, so I don't mind per se.
[me: http://InterviewKickstart.com]
It's a well researched article, so I don't mind per se.
[me: http://InterviewKickstart.com]
> Bemoaning that non-technical people are the first to filter resumes is silly because it’s not going to change. What can change, however, is how they do the filtering. We need to start thinking analytically about these things, and I hope that publishing this data is a step in the right direction.
This is only true for very few companies (massive ones like Google). If you are running and engineering team of < 50 I think it makes a hell of a lot of sense for resume review to be on the engineers. 10 minutes a day of resume reviewing gets you through a ton of resumes (it takes me less than 60 seconds to determine if I want to continue talking to someone from a resume) at very little cost. You don't want your engineers handling scheduling etc., but resume review isn't really much of a time sink until you are getting tons of inbound all the time, which many companies wont ever get to.
And this:
> As you can see, “good” resumes focused much more on action words/doing stuff (“manage”, “ship”, “team”, “create”, and so on) versus “bad” resumes which, in turn, focused much more on details/technologies used/techniques.
Is highly biased by the fact that she was hiring for a web dev company. Resumes including words like "systems", "C++" and "algorithm" were considered bad because they received no offer. You don't really need the distributed systems guy who can write highly performant C++ and actually understands how to apply algorithms at a standard web dev job.
This is only true for very few companies (massive ones like Google). If you are running and engineering team of < 50 I think it makes a hell of a lot of sense for resume review to be on the engineers. 10 minutes a day of resume reviewing gets you through a ton of resumes (it takes me less than 60 seconds to determine if I want to continue talking to someone from a resume) at very little cost. You don't want your engineers handling scheduling etc., but resume review isn't really much of a time sink until you are getting tons of inbound all the time, which many companies wont ever get to.
And this:
> As you can see, “good” resumes focused much more on action words/doing stuff (“manage”, “ship”, “team”, “create”, and so on) versus “bad” resumes which, in turn, focused much more on details/technologies used/techniques.
Is highly biased by the fact that she was hiring for a web dev company. Resumes including words like "systems", "C++" and "algorithm" were considered bad because they received no offer. You don't really need the distributed systems guy who can write highly performant C++ and actually understands how to apply algorithms at a standard web dev job.
I agree. As a hiring manager, I prefer to do resume reviews. It's an activity that can be done with a six pack (or bottle) in the evening, and still be 10x as accurate and efficient as an HR manager. One would think they would have the pattern recognition, but they don't know enough to read between the lines. ("She doesn't have any technologies listed on the resume, and didn't graduate college, but went straight to an engineering job at Apple after 2 years at U of I and has been there for 5 years" is something that a hiring manager would appreciate and an HR manager would miss.)
I've always thought the best resume format is "Action -> Result" rather than simply listing your responsibilities and/or technical knowledge. People get hired based not on what they did or know, but based on how what they did helped add value to their company.
Example:
Example:
WEAK
----
- Responsible for development and maintenance of the FooBar software suite
- Skills applied: Java, HTML, CSS, REST APIs
STRONG
----
- Added BarBuzz feature to FooBar software suite, which contributed to 25% increase in product sales and winning 2 industry awards.I agree, action -> result always sounds more impressive.
But to a point it's always felt disingenuous to me as far as engineering goes. Sure, I implemented the BarBuzz feature. But a marketing person gave the idea, a product person specced it out, a UX person designed it, a UI person made it pretty, and all I did was the final implementation.
Did I really add the feature, or did I just implement it?
But to a point it's always felt disingenuous to me as far as engineering goes. Sure, I implemented the BarBuzz feature. But a marketing person gave the idea, a product person specced it out, a UX person designed it, a UI person made it pretty, and all I did was the final implementation.
Did I really add the feature, or did I just implement it?
Everyone else who touched the feature is going to also say they added it, on their resume. You have as much of a claim to it as they do. Your job title and the size of the company is clear, so the reader can infer the size of your contribution from that.
Well at some point you can say the same thing of all of them right? Nobody thinks you did this on your own.
In a way this is a feature not a bug-- when an HR person whose job was created primarily to boost the number of females and minorities in the company to meet quotas is reviewing resumes and rejects me, she saves me a lot of time working for a terrible company.
I wish Amazon had done that to me.
Also, of course as you point out her definition of "Good" is based on whether or not the resume got past her arbitrary ill-informed ideas of whats "good". Self reinforcing.
I wish Amazon had done that to me.
Also, of course as you point out her definition of "Good" is based on whether or not the resume got past her arbitrary ill-informed ideas of whats "good". Self reinforcing.
The grammar one worries me. I wonder if that was corrected for country of birth, e.g. if this is an apples for apples comparison of native English speakers (USA, Canada, UK, Australia etc) or if it includes non-native speakers.
If there are people of non-English decent, I wonder if they would be more likely to have errors, especially grammatical, that are understandable. If that is the case, this may simply be a correlation to poor interview performance, and a bias towards the singular group of native English speakers, and that they may have let great programmers go.
Now, I'm not saying it is true, I am just saying that is a possibility that would alarm me when errors on a resume is such a huge indicator, and wonder whether that is not something they should be correcting for.
If there are people of non-English decent, I wonder if they would be more likely to have errors, especially grammatical, that are understandable. If that is the case, this may simply be a correlation to poor interview performance, and a bias towards the singular group of native English speakers, and that they may have let great programmers go.
Now, I'm not saying it is true, I am just saying that is a possibility that would alarm me when errors on a resume is such a huge indicator, and wonder whether that is not something they should be correcting for.
Author here. I get this counterpoint a lot and for good reason. My best response is that a resume isn't something you come up with on the fly. It's a document you have every opportunity to show to people, get feedback, and improve upon. Therefore, if you're not a native speaker, you should presumably have the self-awareness to mitigate that disadvantage by enlisting outside help.
I understand those arguments, but it doesn't addresses the issue for a very specific reason. What if the problem is the way internally the team works?
When measuring end-to-end, any part of that may influence the final outcome. If, and I am not saying this is true, if there is a tendency to have a few more errors (especially grammatical with it's its) in a non-native tongue, success outcomes MAY point to a bias in interviewing, or in how people interact internally.
Or not. It is just something I wondered, and something it doesn't seem like you've looked at specifically, and understandably as well, because who wants to open themselves to accusations of bias with their own data?
When measuring end-to-end, any part of that may influence the final outcome. If, and I am not saying this is true, if there is a tendency to have a few more errors (especially grammatical with it's its) in a non-native tongue, success outcomes MAY point to a bias in interviewing, or in how people interact internally.
Or not. It is just something I wondered, and something it doesn't seem like you've looked at specifically, and understandably as well, because who wants to open themselves to accusations of bias with their own data?
I opposed the connection between general grammar/spelling and what people do on the job. I'm a perfect example of it: informal writings are quite rebellious while professional ones are carefully written and revised. Because I'm paid to and misunderstandings cost more. :)
That said, I totally agree with your comment here that the preparation time available should render resumes free of such errors. They should be peer reviewed, as well. Problems caused by a lack of either indicate a character flaw to me.
That said, I totally agree with your comment here that the preparation time available should render resumes free of such errors. They should be peer reviewed, as well. Problems caused by a lack of either indicate a character flaw to me.
> If there are people of non-English decent
decent -> descent.
Muphry's law strikes again. https://en.wikipedia.org/wiki/Muphry%27s_law
decent -> descent.
Muphry's law strikes again. https://en.wikipedia.org/wiki/Muphry%27s_law
And there are differences in acceptable grammar/ writing styles in all of the English speaking country's
The USA is more permissive of double negatives for example.
And this ignores the fact that some candidates will have dyslexia.
The USA is more permissive of double negatives for example.
And this ignores the fact that some candidates will have dyslexia.
If having worked at a top company is more correlated with success than any combination of one's own metrics, then I'd look for ways to improve the recruiting process.
Well, it's maybe an admission that the top company's hiring process is better than your own. Which, if you think about it, might be reasonable.
Still, yes, it's a sign that your own isn't top-company-worthy yet...
Still, yes, it's a sign that your own isn't top-company-worthy yet...
> Well, it's maybe an admission that the top company's hiring process is better than your own
Or maybe the most naturally talented candidates self select into the bigger best known firms from the onset?
Or maybe the most naturally talented candidates self select into the bigger best known firms from the onset?
Or they're suffering from brand recognition. Google and Disney have a reputation for excellence, so just having it on your resume biases the interviewer into projecting those attributes onto you.
Yeah, having worked for a couple of Fortune 500 companies, 1 was fine to work for, the other was absolute hell.
Reasonable if the top company's needs with respect to engineers matches yours. Just because someone was what Google wanted (or thought they wanted) does not mean the same is true for you.
Good point. Google is something of an omnivore. If you're talented, they can hire you and then try to figure out what to do with you. For a smaller company, your needs are often much narrower.
Some comments:
-- If you're running a good company, then the resumes will be filtered by technical people, not HR. There really aren't too many of them (if you're big enough to be getting thousands of resumes, then you have enough engineers to do a first pass.) And don't just give the resumes to these engineers, you need to train them on what to look for.
It is already changing that Engineers filter the resumes and not HR. Good startups do this.
Hiring -- especially for a startup-- is absolutely the most valuable thing, and it is a valuable use of engineers time (but only have engineers who want to do it and care about the hiring process do it.)
"Before I share the actual results, a quick word about context is in order. TrialPay’s hiring standards are quite high. We ended up interviewing roughly 1 in 10 people that applied. "
A non-technical person (yeah yeah, I don't believe for a second she was ever an engineer, that's vanity talking) is eliminating %90 of the applicants and they think that's a "high bar"? No. That's randomness.
-- Top Company.
I love that she thinks having worked at a "Top company" like Amazon is an indicator of success. I pick amazon because I've worked there and its in the news lately for being totally poorly managed. That poor management means the engineering side of the house is a total and absolute mess. Bugs I fixed in 2006 are STILL BROKEN. Because they were regressed due to mismanagement. The QA team that was focusing on that area is totally disbanded. This areas of the site has not improved at all, and has in fact gotten worse over the past 10 years-- and it's critical- it's product search!
So, they will hire people from Top Companies (and put them thru the incompetent HR filter) over better engineers with good side projects.
Great.
It is already changing that Engineers filter the resumes and not HR. Good startups do this.
Hiring -- especially for a startup-- is absolutely the most valuable thing, and it is a valuable use of engineers time (but only have engineers who want to do it and care about the hiring process do it.)
"Before I share the actual results, a quick word about context is in order. TrialPay’s hiring standards are quite high. We ended up interviewing roughly 1 in 10 people that applied. "
A non-technical person (yeah yeah, I don't believe for a second she was ever an engineer, that's vanity talking) is eliminating %90 of the applicants and they think that's a "high bar"? No. That's randomness.
-- Top Company.
I love that she thinks having worked at a "Top company" like Amazon is an indicator of success. I pick amazon because I've worked there and its in the news lately for being totally poorly managed. That poor management means the engineering side of the house is a total and absolute mess. Bugs I fixed in 2006 are STILL BROKEN. Because they were regressed due to mismanagement. The QA team that was focusing on that area is totally disbanded. This areas of the site has not improved at all, and has in fact gotten worse over the past 10 years-- and it's critical- it's product search!
So, they will hire people from Top Companies (and put them thru the incompetent HR filter) over better engineers with good side projects.
Great.
> A non-technical person (yeah yeah, I don't believe for a second she was ever an engineer, that's vanity talking) is eliminating %90 of the applicants and they think that's a "high bar"? No. That's randomness.
I wish more people could wrap their heads around this. Her process could be rolling a D10 for every CV she receives, then lighting the CV on fire unless it passes the saving throw, and she'd still be able to say "we ended up interviewing roughly 1 in 10 people that applied". It says absolutely nothing about the effectiveness of her methods.
I've been involved in hiring in the past, and discussed this issue with people having this mindset, even using this exact example with the D10, but I don't think I've ever gotten through. Some people will insist on believing that any highly selective screening process is automatically good, without any further introspection.
I wish more people could wrap their heads around this. Her process could be rolling a D10 for every CV she receives, then lighting the CV on fire unless it passes the saving throw, and she'd still be able to say "we ended up interviewing roughly 1 in 10 people that applied". It says absolutely nothing about the effectiveness of her methods.
I've been involved in hiring in the past, and discussed this issue with people having this mindset, even using this exact example with the D10, but I don't think I've ever gotten through. Some people will insist on believing that any highly selective screening process is automatically good, without any further introspection.
I just divide the pile in two and throw half the resumes I get in the trash. Why would I want to hire unlucky people? They could make the whole company unlucky. Plus, it makes our hiring process more selective.
In seriousness though, my hypothesis is that they are at least certain that they are not making the candidate pool any worse by their screening, i.e. they are not selecting for unqualified candidates. Even if their process is barely better than 50%+1, they are at least certain it is not worse than 50%, and consider that good enough. If they took 1000 candidates, 100 of which were qualified for the position, and whittled that down to 100 candidates, 11 of which were qualified for the position, they'd honestly consider that a win.
This seems pretty reasonable; "having worked at a top company matters" could speak more to the hiring practices of "top company" than to anything else.
A candidate who worked at a "top company" was (probably) already prefiltered by "top company"'s hiring practices.
A candidate who worked at a "top company" was (probably) already prefiltered by "top company"'s hiring practices.
The same is true from name brand schools. They filter for good test scores and other things.
Holy selection bias Batman!
So the people with resumes so awesome that they could land an interview despite having no CS degree did well in interviews? Color me unimpressed.
So the people with resumes so awesome that they could land an interview despite having no CS degree did well in interviews? Color me unimpressed.
I can't comment on the typo thing as it never seems to be a problem in my team. But here are some thoughts for the hiring culture in China. I am a technical manager and currently taking care of a team of engineers in a large online education company. I do recruiting myself.
> having attended a top computer science school doesn’t matter.
Matters to me. Unlike in U.S., the selection process for college/university entrance is more effective in China, especially for top schools. I am not saying those from less top schools are not as good. It is just more efficient for the resume screening. The education system has done this much better than a short interview or resume.
> listing side projects on your resume isn’t as advantageous as expected.
Somewhat agree. It does have some impact, but I will read the code. It is advantageous for me as I can see his previous work in deep details.
> GPA doesn’t seem to matter.
Hm... here nobody lists GPA unless it's very high, and high GPA means nothing to me.
> having worked at a top company matters.
It does. Former employees from Baidu, Alibaba and Tencent (BAT) are more welcome, for the same reason as the top school thing. The selection there is much more strict and cruel. But I often ask the question 'why you leave B/A/T while they pay better and are generous in stock options?'. The question implies my concern that the candidate leaves B/A/T for some weakness that may also have bad influence in my team.
The branding of top company is a double-edged sword. While you are enjoying the branding, you must be prepared to explain why giving it up.
> having attended a top computer science school doesn’t matter.
Matters to me. Unlike in U.S., the selection process for college/university entrance is more effective in China, especially for top schools. I am not saying those from less top schools are not as good. It is just more efficient for the resume screening. The education system has done this much better than a short interview or resume.
> listing side projects on your resume isn’t as advantageous as expected.
Somewhat agree. It does have some impact, but I will read the code. It is advantageous for me as I can see his previous work in deep details.
> GPA doesn’t seem to matter.
Hm... here nobody lists GPA unless it's very high, and high GPA means nothing to me.
> having worked at a top company matters.
It does. Former employees from Baidu, Alibaba and Tencent (BAT) are more welcome, for the same reason as the top school thing. The selection there is much more strict and cruel. But I often ask the question 'why you leave B/A/T while they pay better and are generous in stock options?'. The question implies my concern that the candidate leaves B/A/T for some weakness that may also have bad influence in my team.
The branding of top company is a double-edged sword. While you are enjoying the branding, you must be prepared to explain why giving it up.
I was an engineering manager in one of the big Internet companies in Shanghai for 2 years.
As a foreigner (Singaporean), my own experience was those graduating from top Chinese schools actually is a dis-service to the whole team since those graduates tend to think too high of themselves, and thus are arrogant, slacking, demanding high benefits (salary, holidays, etc.), but their outputs don't commensurate with their attitude. Definitely there are good graduates from those schools and your experience may differ, but so far this is my observation.
As such I now no longer hire graduates from top Chinese colleges. Instead, those from second or third tier schools are more humble and hardworking.
No, I am not suppressing any salary. My staff (who came from a pretty not-well-known school) got a monthly basic salary of RMB25K, which I think isn't low in any standard, even in Shanghai. But I think he deserved every cent of it and I am happy to pay.
As a foreigner (Singaporean), my own experience was those graduating from top Chinese schools actually is a dis-service to the whole team since those graduates tend to think too high of themselves, and thus are arrogant, slacking, demanding high benefits (salary, holidays, etc.), but their outputs don't commensurate with their attitude. Definitely there are good graduates from those schools and your experience may differ, but so far this is my observation.
As such I now no longer hire graduates from top Chinese colleges. Instead, those from second or third tier schools are more humble and hardworking.
No, I am not suppressing any salary. My staff (who came from a pretty not-well-known school) got a monthly basic salary of RMB25K, which I think isn't low in any standard, even in Shanghai. But I think he deserved every cent of it and I am happy to pay.
Would not the history of imperial exams going back thousands of years be a cause of cultural bias here?
The more I read this, the more skeptical I become. All this shows is "What characteristics are most likely to get you an offer?" This is a little important from a narrow process metric, but isn't valuable to the company's bottom line.
2 more important questions: 1 - "Which characteristics are more likely to appear in high performers than low performers and non-hires?"
2 - "Are we over-weighting or under-weighting certain characteristics in our recruiting process based on our knowledge of question 1?"
Question 2 is actually much harder to answer. For example, if you find no correlation between GPA and performance, it isn't that GPA doesn't matter, it's that you're already weighing it properly in the performance decision. (It could be that you ignore it, or it could be that you give it tons of weight, but either way, the lack of correlation to performance post-hire means you're doing the right thing)
2 more important questions: 1 - "Which characteristics are more likely to appear in high performers than low performers and non-hires?"
2 - "Are we over-weighting or under-weighting certain characteristics in our recruiting process based on our knowledge of question 1?"
Question 2 is actually much harder to answer. For example, if you find no correlation between GPA and performance, it isn't that GPA doesn't matter, it's that you're already weighing it properly in the performance decision. (It could be that you ignore it, or it could be that you give it tons of weight, but either way, the lack of correlation to performance post-hire means you're doing the right thing)
I've read and re-read this post a number of times. It resonates because hiring is broken and companies are looking for ways to triage the process. Yet when I look at the data points being captured, I ask 'what is really being measured?' Ability to work and ship product or Hoop jumpers? (https://news.ycombinator.com/item?id=10073663)
A new hire at a Startup can and should look more like what I found at lighttable, cf 'Meet the new guy': http://lighttable.com/2015/01/13/light-table-hack-night/
The progression from plugin contributor to new hire is measured in actual results and peer interaction, not through second order measurements. Adapt or die.
A new hire at a Startup can and should look more like what I found at lighttable, cf 'Meet the new guy': http://lighttable.com/2015/01/13/light-table-hack-night/
The progression from plugin contributor to new hire is measured in actual results and peer interaction, not through second order measurements. Adapt or die.
>> listing side projects on your resume isn’t as advantageous as expected
I am surprised by this "finding". Why would side projects not be advantageous? Don't side projects establish that the candidate is actively keeping up to date on technologies and is so passionate about it that he/she works on it even outside office hours?
I am surprised by this "finding". Why would side projects not be advantageous? Don't side projects establish that the candidate is actively keeping up to date on technologies and is so passionate about it that he/she works on it even outside office hours?
From the article:
> Here’s what ends up happening. To game the system, applicants start linking to virtually empty GitHub accounts that are full of forked repos where they, at best, fixed some silly whitespace issue. [...] Outside of that, there’s the fact that not all side projects are created equal. [...] While awesome side projects are a HUGE indicator of competence, if the people reading resumes can’t (either because of lack of domain-specific knowledge or because of time considerations) tell the difference between awesome and underwhelming, the signal gets lost in the noise.
It looks like genuinely interesting side projects are a good indicator, but enough people list trivial repos on their resume that just having bullet points on the piece of paper on its own isn't a significant indicator of quality anymore. The only thing she was counting for her analysis is whether or not side projects were listed on the resume, she wasn't taking quality into effect.
> Here’s what ends up happening. To game the system, applicants start linking to virtually empty GitHub accounts that are full of forked repos where they, at best, fixed some silly whitespace issue. [...] Outside of that, there’s the fact that not all side projects are created equal. [...] While awesome side projects are a HUGE indicator of competence, if the people reading resumes can’t (either because of lack of domain-specific knowledge or because of time considerations) tell the difference between awesome and underwhelming, the signal gets lost in the noise.
It looks like genuinely interesting side projects are a good indicator, but enough people list trivial repos on their resume that just having bullet points on the piece of paper on its own isn't a significant indicator of quality anymore. The only thing she was counting for her analysis is whether or not side projects were listed on the resume, she wasn't taking quality into effect.
That forking behavior is exactly why I hate the idea of companies expecting people to use Github to showcase their side projects.
And I really support the idea of side projects, preferably on something other than a default Github page though.
And I really support the idea of side projects, preferably on something other than a default Github page though.
> Don't side projects establish that the candidate is actively keeping up to date on technologies and is so passionate about it that he/she works on it even outside office hours?
Side projects demonstrate that a candidate has other irons in the fire, which is a negative indicator for dependencies on the potential hiring firm. Plus, developers are rarely hourly positions with a well-defined "outside of office hours", outside work may be viewed as a distraction from what they could be doing for the employer.
Side projects demonstrate that a candidate has other irons in the fire, which is a negative indicator for dependencies on the potential hiring firm. Plus, developers are rarely hourly positions with a well-defined "outside of office hours", outside work may be viewed as a distraction from what they could be doing for the employer.
Side projects are probably much more advantageous if you get to talk about them in an actual interview. I suspect they don't matter to much on resumes because of the issues others have brought up here
I disagree with the author too. Interviewers are very excited to talk to me about my side projects. I'm a pretty cynical person but I don't think they are doing it to trick me.
The quote was specifically about the value on resumes. The article was written from the perspective of choosing who to interview in the first place. It may be that your resume was selected because the interviewers were excited to talk about your side projects, but that assumes the interviewers were the ones selecting candidates for interviews.
https://news.ycombinator.com/item?id=6326477