Coding Interviews Are BS (largely)(coscreen.co)
coscreen.co
Coding Interviews Are BS (largely)
https://www.coscreen.co/blog/you-are-interviewing-developers-all-wrong/
28 comments
Great, if not for the healthcare and retirement savings being tied to employment. Having left a role after only 4 months my SO is not happy with so many changes.
Getting rid of healthcare and retirement savings being tied to employment is good for a variety of reasons - this is just one of them.
In fact, I'm not sure why anyone would want to keep them tied to employment.
In fact, I'm not sure why anyone would want to keep them tied to employment.
Employers like it for retention
I don't think even the employers like it. It costs a lot and it's a pain in the ass to even sign up for if you're a small company. To add an insult to an injury, costs go up every year like clockwork in a largely out-of-control fashion. I worked for several companies now which started from scratch. "Retention" was never a part of the conversation, but "pain in the ass" always was. I think employers provide health insurance simply because employees expect it, as well as to pay you less than they would have to pay you otherwise. It's tax advantaged on their end. If your accountant is any good, it can even be tax advantaged for single person LLCs. Most people don't know this, though. Why this is not tax deductible for "regular" people I have no idea. IMO it should either be tax deductible for everyone, or no one. To have this in-between situation is idiotic.
That's not really the employers problem. I'm surprised more companies don't do this, at least in America where there seems little legislation in the way.
In one of Rick Atkinson's trilogy on the US Army in WW II, some general is quoted as remarking that under WW II rules, Robert E. Lee and some other prominent general--Grant or Sherman would probably qualify for the example--would have been removed from command and sent off to training duties.
Jason Thomas here, CTO at CoScreen. Deep inside I had always suspected coding interviews were in large part bullshit.
Originally I based my ambitions on the collective wisdom of the Internet (what could go wrong?), and of course looked towards the FANG interview process, and especially Google.
I was the first full-time engineer hired at the video interviewing pioneer HireVue, and have a decade of experience in the HR and interviewing technology space. This experience made me a firm believer in the structured interview. Line up a set of candidates, give them a set of standard questions in a structured fashion and look for genius in the answer of routine questions.
What could be fairer and more organized than lining up candidates one by one, question by question, rating each, and just hiring the best rated out of all?
Read my post for what I learned and how to set up a truly fair and objective interviewing framework.
Originally I based my ambitions on the collective wisdom of the Internet (what could go wrong?), and of course looked towards the FANG interview process, and especially Google.
I was the first full-time engineer hired at the video interviewing pioneer HireVue, and have a decade of experience in the HR and interviewing technology space. This experience made me a firm believer in the structured interview. Line up a set of candidates, give them a set of standard questions in a structured fashion and look for genius in the answer of routine questions.
What could be fairer and more organized than lining up candidates one by one, question by question, rating each, and just hiring the best rated out of all?
Read my post for what I learned and how to set up a truly fair and objective interviewing framework.
No offense, but what could be fairer than offering college admissions to people based on lining them up and having them take a standardized test?
Honesty, there is no perfect solution. There’s a better solution than the status quo but there is no “fairest solution because there are 68,000 ways to define fair
Edit: uggg I fell for the “bait”
Anyway, could you point to the year when the collective wisdom of hacker news or even the “internet” considered coding interviews like googles or leet code to be good and not the subject of universal ire? I don’t ever remember that time so I’m confused by the tone of the comment with its points like:
>“ Originally I based my ambitions on the collective wisdom of the Internet (what could go wrong?), and of course looked towards the FANG interview process, and especially Google.”
Honesty, there is no perfect solution. There’s a better solution than the status quo but there is no “fairest solution because there are 68,000 ways to define fair
Edit: uggg I fell for the “bait”
Anyway, could you point to the year when the collective wisdom of hacker news or even the “internet” considered coding interviews like googles or leet code to be good and not the subject of universal ire? I don’t ever remember that time so I’m confused by the tone of the comment with its points like:
>“ Originally I based my ambitions on the collective wisdom of the Internet (what could go wrong?), and of course looked towards the FANG interview process, and especially Google.”
Jason, CTO of CoScreen here on my normal account.
So, this is about coding interviews for hiring developers, and it's more about the goal of hiring a good engineer.
Obviously evaluating them dynamically will be more heuristic, and less fair, but it has worked far better for me than using standard uniform async interviews.
So, this is about coding interviews for hiring developers, and it's more about the goal of hiring a good engineer.
Obviously evaluating them dynamically will be more heuristic, and less fair, but it has worked far better for me than using standard uniform async interviews.
Orthogonal to the app being a cool way to do interviews, am I the only one perplexed by the example this person chose to highlight problems with coding interviews?
If I understand correctly, they hired someone who seemed to have a good grasp of algorithms but when it came time to code, took forever to write code and took forever to type something out?
Isn't that the exact thing that code interviews are good for? Finding out if someone can write legible, working code quickly?
Code interviews are terrible for finding out if someone can do deep thinking on a very difficult problem over a long period of time, but the exact problem scenario this hire had should have been caught.
I'm not sure how having them write/debug some slightly more complex code in IDEA vs an online Vim approximation is going to suddenly keep someone similar to the 'candidate' referenced from slipping through.
I'd also argue that bootstrapping a project is very different than managing a great code base. I'd be very skeptical of the idea that evaluating someone's toy project tells you how well they'll do in a massive code base.
If I understand correctly, they hired someone who seemed to have a good grasp of algorithms but when it came time to code, took forever to write code and took forever to type something out?
Isn't that the exact thing that code interviews are good for? Finding out if someone can write legible, working code quickly?
Code interviews are terrible for finding out if someone can do deep thinking on a very difficult problem over a long period of time, but the exact problem scenario this hire had should have been caught.
I'm not sure how having them write/debug some slightly more complex code in IDEA vs an online Vim approximation is going to suddenly keep someone similar to the 'candidate' referenced from slipping through.
I'd also argue that bootstrapping a project is very different than managing a great code base. I'd be very skeptical of the idea that evaluating someone's toy project tells you how well they'll do in a massive code base.
Jason (article writer) here on my normal ycombinator account.
> but the exact problem scenario this hire had should have been caught.
I think if people weren't taking courses specifically training for engineering interviews, and common interview challenges, maybe this wouldn't be a problem. The truth is gamified programming challenges (commonly used for interviews) do not have a strong relationship to actual day to day programming skills, like reading code, understanding documentation, and doing research.
I suppose I'm just reiterating your point, but I'm not sure why you come to a different conclusion regarding the person I hired being slow reading our code, doing research, and setting up basic tooling being something such an interview process would weed out.
It didn't, and hasn't like I said, any more than a coin flip, in my personal (anecdotal of course) experience.
> I'm not sure how having them write/debug some slightly more complex code in IDEA vs an online Vim approximation is going to suddenly catch people like this?
It won't, if that's all I was suggesting.
I was suggesting good engineers may be being filtered due to duress in the interview being caused by poor tooling.
The real focus was having them do a small semi-real project of some kind, and then having a solid engineer pair-program with them.
This has proven to be a successful strategy for us.
> but the exact problem scenario this hire had should have been caught.
I think if people weren't taking courses specifically training for engineering interviews, and common interview challenges, maybe this wouldn't be a problem. The truth is gamified programming challenges (commonly used for interviews) do not have a strong relationship to actual day to day programming skills, like reading code, understanding documentation, and doing research.
I suppose I'm just reiterating your point, but I'm not sure why you come to a different conclusion regarding the person I hired being slow reading our code, doing research, and setting up basic tooling being something such an interview process would weed out.
It didn't, and hasn't like I said, any more than a coin flip, in my personal (anecdotal of course) experience.
> I'm not sure how having them write/debug some slightly more complex code in IDEA vs an online Vim approximation is going to suddenly catch people like this?
It won't, if that's all I was suggesting.
I was suggesting good engineers may be being filtered due to duress in the interview being caused by poor tooling.
The real focus was having them do a small semi-real project of some kind, and then having a solid engineer pair-program with them.
This has proven to be a successful strategy for us.
>I was suggesting good engineers may be being filtered due to duress in the interview being caused by poor tooling.
Ah, well said, and I agree. If I don't have Vim keys I'm about 5x slower, so I can imagine how someone used to a certain setup is equally handicapped in an artificially constrained interview process.
Ah, well said, and I agree. If I don't have Vim keys I'm about 5x slower, so I can imagine how someone used to a certain setup is equally handicapped in an artificially constrained interview process.
Coding assessments are largely used to filter unqualified candidates and save time. You can't reasonably interview every applicant, so you set a small barrier in place and those who can cross it get the time. Coding in interviews is more about understanding your logical decisions and brain flow, whether you ask the right questions or approach problems in a reasonable way.
Full disclosure, we still use hackerrank for just this internally.
I just think if a candidate scores a 70%, I don't expect them to be better or worse than a candidate who scores 100%.
Having criteria to perform an interview is one thing, but if your interview is just to go in with another toy challenge on a whiteboard, I think it would be much better to do a demo project, and immediately pair-program before making the decision to hire.
I just think if a candidate scores a 70%, I don't expect them to be better or worse than a candidate who scores 100%.
Having criteria to perform an interview is one thing, but if your interview is just to go in with another toy challenge on a whiteboard, I think it would be much better to do a demo project, and immediately pair-program before making the decision to hire.
Someone called this to my attention in terms of making engineering interviews more fair, and there are some overlapping points, particularly with not biasing your interviews based on tooling:
https://faircoding.com/interviews/
https://faircoding.com/interviews/
This is bullshit. Let me explain why.
The author posits that the best way to interview is to simulate an actual work environment for the candidate as if he was working in an actual code environment. In theory, this would be good. After all, the best way to understand a candidate is to see how he actually works.
The reality is that the Leetcode style coding interview, as we know it today, is best approached the same way you would approach the engineering process in general: prototype, refine the idea, test the idea, revise based on inefficiencies, code it. Any semi-competent moron can do this if he gives a damn about his career because this is actually what people should be doing day to day: thinking about how and why the approach to code would be effective. This is logical thinking that can take years to perfect and adjust, especially when your career progression means tackling increasingly difficult problem areas (from deterministic products to nondeterministic infrastructure issues).
Which is why most of the internet thinks the interview is BS: because they think like code monkies, not engineers. Most people believe software engineering is copy-pasting stuff from Github and then moving on when its using software and code to solve a problem.
In the author's eyes, he thinks that more weight should be given to the soft skills. Soft skills that can easily be learned on the job. I mean, how long does it take for someone to dive into the code? Just follow the history of changes of that line. A day or 2 of practice. How long does it take for someone to learn new tools like ripgrep? 2 days. These are things that can easily be picked up with less than a year of industrial programming.
Which is effectively what onboarding is: getting a person used to the tools, nuances, and implicit culture of a company.
And different companies have different needs. When you're a startup, you need to code fast and quick, ship the product, and itterate. In a larger corporate environment, you need to code intelligently, be careful of the leverage you are given, and truly understand what you're writing.
In each of the cases, the dimensions you're judged against are going to be different. You artificially restrict your own candidate pool to culture fits instead of people who can think like actual engineers. For big tech companies, it doesn't matter: they have so many people coming through the door that they couldn't care less. If anything, this hurts startups (ironically) because now they have a smaller group of people to work with. Or a larger one if you're dismissing all the people who can actually thinking like a proper engineer.
I mean, if a startup wants to hire someone who says "I don't know why the code works, I copied it from stack overflow and moved on with my day" then by all means, have him write the Robinhood options platform.
Ceter paribus, I would take the former candidate: someone who's clear in logical thinking, then ram him into reality and the company until he learns. At least he would learn quicker. On the other hand, it would take someone who's the opposite, it would take me at least 5 years for him to even begin to understand to not be a code monkey.
Secondly, the Leetcode style interview is actually scalable. Any engineer can give it and can evaluate the problem solving strategies that the candidate uses. That's what allows companies to be efficient with their hiring: people are interchangeable. On the other hand, doing something as nebulous as judging the soft skills of engineering is going to be way too subjective and give more noise-to-signal.
Example: if an infrastructure engineer is judging a product engineer, there's going to be a big disconnect considering that the problem space of infra engineers is more nebulous and the solution is more unknown. A product engineer would shit his pants if he was asked how to migrate a codebase at scale.
Of course, you could get around this by standardizing the test and pairing engineers with engineers of similar backgrounds. But along what dimensions would you standardize? What if the candidate can solve the issue without tracing the code history? How can you create a test where a candidate has to go through the code history without the answer being blatantly telegraphed (ie. when did this bug occur?)? And how could you do this effectively at scale without screwing over your own company's resources (again, think startups. Only so many engineers to go around.)
A more honest way to achieve the same thing (if the author truly wanted to see if the candidate was well suited for the company) without going into Leetcode would be the company to put a probationary period on new-hires. It may cost more but you might as well filter hard and fast instead of using a easy/wrong filter simply because a bunch of newgrad Reddit idiots who can't pass a coding interview are crying their collective heads off about it being unfair.
But if a bunch of smart and successful engineers can consistently land offers, then why can't other smart and successful engineers? Even Max Howell, creator of homebrew, who flamed the coding interview for being shit (his tweet about Homebrew and inverting a binary tree), still got an Senior Apple offer. So the fact of the matter is that it works. It might not be efficient, yes. But you cannot deny it works.
TL;DR: Author is full of shit about Leetcode interviews and his thesis would actually screw over smaller companies.
The author posits that the best way to interview is to simulate an actual work environment for the candidate as if he was working in an actual code environment. In theory, this would be good. After all, the best way to understand a candidate is to see how he actually works.
The reality is that the Leetcode style coding interview, as we know it today, is best approached the same way you would approach the engineering process in general: prototype, refine the idea, test the idea, revise based on inefficiencies, code it. Any semi-competent moron can do this if he gives a damn about his career because this is actually what people should be doing day to day: thinking about how and why the approach to code would be effective. This is logical thinking that can take years to perfect and adjust, especially when your career progression means tackling increasingly difficult problem areas (from deterministic products to nondeterministic infrastructure issues).
Which is why most of the internet thinks the interview is BS: because they think like code monkies, not engineers. Most people believe software engineering is copy-pasting stuff from Github and then moving on when its using software and code to solve a problem.
In the author's eyes, he thinks that more weight should be given to the soft skills. Soft skills that can easily be learned on the job. I mean, how long does it take for someone to dive into the code? Just follow the history of changes of that line. A day or 2 of practice. How long does it take for someone to learn new tools like ripgrep? 2 days. These are things that can easily be picked up with less than a year of industrial programming.
Which is effectively what onboarding is: getting a person used to the tools, nuances, and implicit culture of a company.
And different companies have different needs. When you're a startup, you need to code fast and quick, ship the product, and itterate. In a larger corporate environment, you need to code intelligently, be careful of the leverage you are given, and truly understand what you're writing.
In each of the cases, the dimensions you're judged against are going to be different. You artificially restrict your own candidate pool to culture fits instead of people who can think like actual engineers. For big tech companies, it doesn't matter: they have so many people coming through the door that they couldn't care less. If anything, this hurts startups (ironically) because now they have a smaller group of people to work with. Or a larger one if you're dismissing all the people who can actually thinking like a proper engineer.
I mean, if a startup wants to hire someone who says "I don't know why the code works, I copied it from stack overflow and moved on with my day" then by all means, have him write the Robinhood options platform.
Ceter paribus, I would take the former candidate: someone who's clear in logical thinking, then ram him into reality and the company until he learns. At least he would learn quicker. On the other hand, it would take someone who's the opposite, it would take me at least 5 years for him to even begin to understand to not be a code monkey.
Secondly, the Leetcode style interview is actually scalable. Any engineer can give it and can evaluate the problem solving strategies that the candidate uses. That's what allows companies to be efficient with their hiring: people are interchangeable. On the other hand, doing something as nebulous as judging the soft skills of engineering is going to be way too subjective and give more noise-to-signal.
Example: if an infrastructure engineer is judging a product engineer, there's going to be a big disconnect considering that the problem space of infra engineers is more nebulous and the solution is more unknown. A product engineer would shit his pants if he was asked how to migrate a codebase at scale.
Of course, you could get around this by standardizing the test and pairing engineers with engineers of similar backgrounds. But along what dimensions would you standardize? What if the candidate can solve the issue without tracing the code history? How can you create a test where a candidate has to go through the code history without the answer being blatantly telegraphed (ie. when did this bug occur?)? And how could you do this effectively at scale without screwing over your own company's resources (again, think startups. Only so many engineers to go around.)
A more honest way to achieve the same thing (if the author truly wanted to see if the candidate was well suited for the company) without going into Leetcode would be the company to put a probationary period on new-hires. It may cost more but you might as well filter hard and fast instead of using a easy/wrong filter simply because a bunch of newgrad Reddit idiots who can't pass a coding interview are crying their collective heads off about it being unfair.
But if a bunch of smart and successful engineers can consistently land offers, then why can't other smart and successful engineers? Even Max Howell, creator of homebrew, who flamed the coding interview for being shit (his tweet about Homebrew and inverting a binary tree), still got an Senior Apple offer. So the fact of the matter is that it works. It might not be efficient, yes. But you cannot deny it works.
TL;DR: Author is full of shit about Leetcode interviews and his thesis would actually screw over smaller companies.
> This is bullshit. Let me explain why.
Hi, Jason here, CTO of CoScreen, and author of this article.
Thanks for your candid opinion Richard.
> The reality is that the Leetcode style coding interview, as we know it today...
I'd love a guide, or other details about how to best pose a "Leetcode" style interview, and refine it. It sounds like a lot of work, but I'm sharing my personal experience interviewing engineers over the last 20+ years. I don't pose just hackerrank like challenges as the only method, but also whiteboard/and ad-hoc in person interviews.
I'm willing to believe there are ways to get better results using these async type methods. But, for my last several companies, including being in management/hiring positions for several startups, I have found them to be very poor at predicting on the job performance.
> In the author's eyes, he thinks that more weight should be given to the soft skills.
I never stated this. I'm not sure where you are getting this from.
Being able to read code isn't a soft skill, nor is being able to navigate large code bases, debug, read documentation, and research. Absolutely these are skills that you can learn on the job, but not by any means easily.
> And different companies have different needs. When you're a startup, you need to code fast and quick, ship the product, and iterate.
I definitely agree with this.
> That's what allows companies to be efficient with their hiring: people are interchangeable.
Have you been working for software companies where you found engineers immediately fungible and replaceable?
This is really different than my own personal experience.
> And how could you do this effectively at scale without screwing over your own company's resources (again, think startups. Only so many engineers to go around.)
My experience hiring has primarily been for startups, and in both Director/CTO, and other positions, have had to spend an inordinate amount of time hiring developers. Scaling to three, or four engineering hires isn't difficult for a startup.
The cost on the other hand, to the point I made in this article, of making those two or three hires bad hires, is extremely high.
> without going into Leetcode would be the company to put a probationary period on new-hires.
Most good programmers are gainfully employed, and won't agree to a probationary period when being hired... Why would they leave a stable role for one that is tentative --- I wouldn't...
I think the onboarding project (could be paid/contracted) like I suggested in the article, as well as a pair-programming session, is what I am suggesting here. You could think of it as a commitment free probationary period on both sides, without requiring the engineer leave their current position.
> But if a bunch of smart and successful engineers can consistently land offers, then why can't other smart and successful engineers?
I'm not sure, really, I'm just offering advice for startups, like mine, that is earnest and based on my own experience.
>Leetcode interviews and his thesis would actually screw over smaller companies.
I don't really mention leetcode once in my article, and mention other forms of traditional interviews--- like in person whiteboard interviews... I call out doing hackerrank challenges.
I'm not sure why you're so focused on leetcode.
If I would have based my hiring on hackerrank alone, I wouldn't have hired one of the best engineers I have worked with in my career.
These coding challenges really don't prove competency in software engineering, again in my personal opinion, based on a large amount of experience in this area.
It's fine for you to disagree with that, but as the Dude would say: that's your opinion man.
Hi, Jason here, CTO of CoScreen, and author of this article.
Thanks for your candid opinion Richard.
> The reality is that the Leetcode style coding interview, as we know it today...
I'd love a guide, or other details about how to best pose a "Leetcode" style interview, and refine it. It sounds like a lot of work, but I'm sharing my personal experience interviewing engineers over the last 20+ years. I don't pose just hackerrank like challenges as the only method, but also whiteboard/and ad-hoc in person interviews.
I'm willing to believe there are ways to get better results using these async type methods. But, for my last several companies, including being in management/hiring positions for several startups, I have found them to be very poor at predicting on the job performance.
> In the author's eyes, he thinks that more weight should be given to the soft skills.
I never stated this. I'm not sure where you are getting this from.
Being able to read code isn't a soft skill, nor is being able to navigate large code bases, debug, read documentation, and research. Absolutely these are skills that you can learn on the job, but not by any means easily.
> And different companies have different needs. When you're a startup, you need to code fast and quick, ship the product, and iterate.
I definitely agree with this.
> That's what allows companies to be efficient with their hiring: people are interchangeable.
Have you been working for software companies where you found engineers immediately fungible and replaceable?
This is really different than my own personal experience.
> And how could you do this effectively at scale without screwing over your own company's resources (again, think startups. Only so many engineers to go around.)
My experience hiring has primarily been for startups, and in both Director/CTO, and other positions, have had to spend an inordinate amount of time hiring developers. Scaling to three, or four engineering hires isn't difficult for a startup.
The cost on the other hand, to the point I made in this article, of making those two or three hires bad hires, is extremely high.
> without going into Leetcode would be the company to put a probationary period on new-hires.
Most good programmers are gainfully employed, and won't agree to a probationary period when being hired... Why would they leave a stable role for one that is tentative --- I wouldn't...
I think the onboarding project (could be paid/contracted) like I suggested in the article, as well as a pair-programming session, is what I am suggesting here. You could think of it as a commitment free probationary period on both sides, without requiring the engineer leave their current position.
> But if a bunch of smart and successful engineers can consistently land offers, then why can't other smart and successful engineers?
I'm not sure, really, I'm just offering advice for startups, like mine, that is earnest and based on my own experience.
>Leetcode interviews and his thesis would actually screw over smaller companies.
I don't really mention leetcode once in my article, and mention other forms of traditional interviews--- like in person whiteboard interviews... I call out doing hackerrank challenges.
I'm not sure why you're so focused on leetcode.
If I would have based my hiring on hackerrank alone, I wouldn't have hired one of the best engineers I have worked with in my career.
These coding challenges really don't prove competency in software engineering, again in my personal opinion, based on a large amount of experience in this area.
It's fine for you to disagree with that, but as the Dude would say: that's your opinion man.
Essentially, the answer is the hiring methodology. If someone seemed like a best fit for a job then they were hired. They then had two weeks to demonstrate basic functional capacity and six weeks to make significant progress. If at any point things did not work out then they were immediately dismissed and replaced with someone else. The high rate of this process meant that there were always open opportunities even for those who had been dismissed from other positions, so things worked out for both employers and employees.