Take-home vs. whiteboard coding: The problem is bad interviews(andrewrondeau.com)
andrewrondeau.com
Take-home vs. whiteboard coding: The problem is bad interviews
https://andrewrondeau.com/blog/2020/04/take-home-vs-whiteboard-coding-the-problem-is-bad-interviews
110 コメント
> I never found reviewing take home interviews to be a great indicator of skill level. It’s hard to know how long was really spent on it, and how much the candidate used resources etc to guide them. Are they a great candidate that cut corners, or a less experienced candidate that used stackoverflow + 10 extra hours to get to a final result.
If you just evaluate them based on their code you doing it wrong. You should organize an in-person code review. Where you ask: Why they do what? How would they improve their code? If something is not right, point out. Look at them how they react. Good candidates usually try to fix it. Bad team players usually defend their implementation until the end. It really easy to find out if they just copy-pasted some code without understanding it.
If you just evaluate them based on their code you doing it wrong. You should organize an in-person code review. Where you ask: Why they do what? How would they improve their code? If something is not right, point out. Look at them how they react. Good candidates usually try to fix it. Bad team players usually defend their implementation until the end. It really easy to find out if they just copy-pasted some code without understanding it.
Take home interviews should be time-limited and proctored real-time. If it's more than 4-6 hours then it's too long. The problem should have multiple parts and just difficult enough to make it unlikely for most candidates to complete everything. The problem should unique enough and broad enough to allow the candidate to demonstrate good code structure, problem solving, creativity, and algorithmic ability. It should obviously not be something so straightforward and simple as to be a question on stack overflow.
The candidate should be encouraged to submit early and often for feedback, and to not be afraid to ask questions. Sometimes a subtle or complex element of the problem statement can help spur this interaction.
Finally the grading should be a blend of both objective criteria such as a series of heuristics and unit tests, as well as subjective measures such as code cleanliness and structure.
Multiple people should be involved in the grading, and a repository of previous submissions should be compared against for consistency.
I and my team used this technique for years to great success, hiring some of the world's top talent.
The candidate should be encouraged to submit early and often for feedback, and to not be afraid to ask questions. Sometimes a subtle or complex element of the problem statement can help spur this interaction.
Finally the grading should be a blend of both objective criteria such as a series of heuristics and unit tests, as well as subjective measures such as code cleanliness and structure.
Multiple people should be involved in the grading, and a repository of previous submissions should be compared against for consistency.
I and my team used this technique for years to great success, hiring some of the world's top talent.
> just difficult enough to make it unlikely for most candidates to complete everything.
How do you judge someone's half-finished work? Do you expect me to write half-finished code on the job?
How do you judge someone's half-finished work? Do you expect me to write half-finished code on the job?
I assume they mean there are individually relatively self-contained pieces of functionality in the problem statement. Not completing an additional feature would still leave the rest of the project in a good state. Perhaps if you use a vcs you could even leave WIP code in an unmerged branch.
My question still stands. How do you evaluate someone's half-finished work? Do you prefer if I finish more features or my already finished code is better quality or better tested?
It's just an unnecessary source of stress during the test.
It's just an unnecessary source of stress during the test.
It would probably be best if someone was available to answer these kinds of questions for the interviewee when their time period starts. Though I think it could also be interesting to have this be part of the assessment -- how does the candidate trade off between scope / quality when they are given a fixed time period?
It is as interesting as evaluating them based on how they indent their code. I can do both but if you don't tell me what you prefer how can I choose?
Re: time spent on take home
Take homes can & ought to be timed.
I've had the opportunity to do this fornat twice and I think it works well.
Here's how it works: They ask you to pick a 3 hour time window to work on the project. You pick the time. They send you the project prompt in an email right at the beginning of your time interval. You have 3 hours to read the prompt, design, implement, test a solution. If you miss the deadline, project counts for nothing.
Both the times I did this the prompts were absolutely doable in 3 hours. They were the kind thing you could do in about 2 hours if you had the ability to read the project prompt ahead of time and think about it for a few days.
Personally, I work well with time constraints. This format feels closer to a workday where mid-afternoon you might challenge yourself to finish a task before 5pm. Its a lot less pressure than having to juggle a conversation and having your code scrutinized in real time.
It forces a real time constraint that keeps you from piling time into something you just don't know.
Its hard to cheat, and it also forces the company to make the task achievable in the time frame. This format forces them to realize when the task is too big for anyone to finish in the time.
Take homes can & ought to be timed.
I've had the opportunity to do this fornat twice and I think it works well.
Here's how it works: They ask you to pick a 3 hour time window to work on the project. You pick the time. They send you the project prompt in an email right at the beginning of your time interval. You have 3 hours to read the prompt, design, implement, test a solution. If you miss the deadline, project counts for nothing.
Both the times I did this the prompts were absolutely doable in 3 hours. They were the kind thing you could do in about 2 hours if you had the ability to read the project prompt ahead of time and think about it for a few days.
Personally, I work well with time constraints. This format feels closer to a workday where mid-afternoon you might challenge yourself to finish a task before 5pm. Its a lot less pressure than having to juggle a conversation and having your code scrutinized in real time.
It forces a real time constraint that keeps you from piling time into something you just don't know.
Its hard to cheat, and it also forces the company to make the task achievable in the time frame. This format forces them to realize when the task is too big for anyone to finish in the time.
I hate time constraints. That is why I hate whiteboarding as well.
My brain just not work that way. If I have to modify something in a project I already worked on. That's fine. But interviews always ask tho build something from the ground up.
Once I got the requirements I need a few hours to think about it, before I write any actual code. I usually do something else while I do this. On a job, it's when I home and do some chores and my mind can wander around freely. But I can't make this process much faster. If I'm on the clock I have to sit down and start spitting out code. Also, there is no time for any major refactoring if you went in a wrong direction. This adds a lot of unnecessary stress.
I like it when you have a day or an afternoon to finish a 3 hours long task. I work faster if there's no time pressure.
My brain just not work that way. If I have to modify something in a project I already worked on. That's fine. But interviews always ask tho build something from the ground up.
Once I got the requirements I need a few hours to think about it, before I write any actual code. I usually do something else while I do this. On a job, it's when I home and do some chores and my mind can wander around freely. But I can't make this process much faster. If I'm on the clock I have to sit down and start spitting out code. Also, there is no time for any major refactoring if you went in a wrong direction. This adds a lot of unnecessary stress.
I like it when you have a day or an afternoon to finish a 3 hours long task. I work faster if there's no time pressure.
The real world has time constraints, so I don't see this as unreasonable expectation in an interview.
> I need a few hours to think...before I write any actual code...I work faster if there's no time pressure.
It sounds like you work better but not faster. If something absolutely must be done in 3 hours, you usually will not have 2 hours to do the dishes while you let it marinate.
To be clear, I prefer to let a problem sit in my head for a bit, too. I think the end result is better in most cases, for most people. But the real world has problems with hard and fast time constraints, or huge financial incentives to get something working in ASAP and clean it up later if necessary.
> I need a few hours to think...before I write any actual code...I work faster if there's no time pressure.
It sounds like you work better but not faster. If something absolutely must be done in 3 hours, you usually will not have 2 hours to do the dishes while you let it marinate.
To be clear, I prefer to let a problem sit in my head for a bit, too. I think the end result is better in most cases, for most people. But the real world has problems with hard and fast time constraints, or huge financial incentives to get something working in ASAP and clean it up later if necessary.
But how many of those involve building a brand new thing you've never thought about from scratch? As the person you responded to said, if they're modifying something they've already worked on, that's different.
If something /brand new/ needs to be built in HOURS with huge financial incentives, that is a failure on so many levels of business that blaming the engineer for their inability to actually get it out the door in 3 hours is ridiculous
If something /brand new/ needs to be built in HOURS with huge financial incentives, that is a failure on so many levels of business that blaming the engineer for their inability to actually get it out the door in 3 hours is ridiculous
Yes, I completely fine with actual hard time pressure, when I know the codebase. When I'm on call and something goes wrong I can fix the code very fast and reliably.
When I presented with a completely new problem I have to think about what is the best tool for the job, how I want to structure my code, how can I test it, what are the edge cases, etc.
When I presented with a completely new problem I have to think about what is the best tool for the job, how I want to structure my code, how can I test it, what are the edge cases, etc.
Yes. Though in practice I'd give people some grace period after the three hours: the three hour time limit is to help the candidate not commit endless hours to the project, it's not meant as a punishment.
I agree here - given the pressures of an interview are much higher than on the job, enforcing a time limit should only prevent extremes rather than add yet more pressure.
I think I’d be fine if someone gave me 3 hours to complete a typical take home exercise, but I worry that some exceptional engineers I know might not be. Applying excessive time pressure could lead to cultural homogeneity.
I think I’d be fine if someone gave me 3 hours to complete a typical take home exercise, but I worry that some exceptional engineers I know might not be. Applying excessive time pressure could lead to cultural homogeneity.
Yup I've done a couple interviews like that. They send the email at the time of your choosing and you have to submit the finished challenge at whatever deadline. Works for me! I can do things how I want to, just as if I had been given such a task on the job.
> They send you the project prompt in an email right at the beginning of your time interval
Whoops, your home internet has gone down. Or Gmail has decided it's not having any mail from their domain. Or your mail provider think they're spam. Or your mail client doesn't pick up the email for half an hour. There's a whole raft of problems coming from "email delivery is reliable enough to be a timer."
Whoops, your home internet has gone down. Or Gmail has decided it's not having any mail from their domain. Or your mail provider think they're spam. Or your mail client doesn't pick up the email for half an hour. There's a whole raft of problems coming from "email delivery is reliable enough to be a timer."
So then you resolve those issues and go on the attempt #2.
I'm not looking gor a dev that knows it all by heart. I'm looking for a dev who knows how to get information about stuff he doesn't know.
A take home interview therefore gives me a much better picture of the real-life skill level of a candidate compared to a very stressful situation like a real-time whiteboard interview.
Also: Give your candidate a real problem to solve and pay them for their time!
A take home interview therefore gives me a much better picture of the real-life skill level of a candidate compared to a very stressful situation like a real-time whiteboard interview.
Also: Give your candidate a real problem to solve and pay them for their time!
There's a famous quote that is attributed to Albert Einstein: “Never memorise something that you can look up.” It is said to be related to a time when a colleague asked him for his phone number, and he reached for his telephone directory to look it up.
At an interview, Albert Szent-Györgyi states basically the same: "You shouldn't fill your head with "szecska" (chaff?). You should keep knowledge in books where you can look it up and use your head only for thinking."
The whole interview was inspiring. It's about learning and exploration. Worth watching if you understand Hungarian: https://youtu.be/rtqhA4TtBrI
The whole interview was inspiring. It's about learning and exploration. Worth watching if you understand Hungarian: https://youtu.be/rtqhA4TtBrI
> are they a great candidate that cut corners, or a less experienced candidate that used stackoverflow
Strange reasoning. If you want to cut corners, you use stack overflow.
Unless you know by heart the optimal way on how to reverse a string in C#, you do a 3 second google search and do a 5 sec copy paste of the one liner, from the top answer on stack overflow.
When I was programming as a 14 year old and came across a problem, it sometimes took me a day to figure something out. There was no internet back then. Now the most obscure issue is probably nicely documented with a top 5 of solutions.
Strange reasoning. If you want to cut corners, you use stack overflow.
Unless you know by heart the optimal way on how to reverse a string in C#, you do a 3 second google search and do a 5 sec copy paste of the one liner, from the top answer on stack overflow.
When I was programming as a 14 year old and came across a problem, it sometimes took me a day to figure something out. There was no internet back then. Now the most obscure issue is probably nicely documented with a top 5 of solutions.
IMO, using stackoverflow is closer to the opposite of cutting corners. No sense laboring to create something that is a 45 second Google search away.
I agree with googling stuff but this gives me anxiety, even though I can create relational databases, use ORM/stored procedures/queries, write web apps, deploy to IIS/Docker. I still worry about looking for the next gig. I did write a complete mission critical solution for a mid-size business, used by 100+ employees but then I am asked about algorithms and I could answer 3 years ago but now I would get stuck on it.
Also, when I interviewed a year ago I was asked to find if something is a palindrome. This is programming 101 and it caught me off-guard. I got so used to all the methods available in C#, that going back to pen/paper and writing a simple for loop felt odd.
Also, when I interviewed a year ago I was asked to find if something is a palindrome. This is programming 101 and it caught me off-guard. I got so used to all the methods available in C#, that going back to pen/paper and writing a simple for loop felt odd.
There are two good reasons to look things up on the internet:
1. Obviously, its faster. I, personally, get paid for results, not for how "clever" I am. (Which is good, 'cause I'm not very clever.)
2. Frequently, the answer on the internet is better/more complete/tested. Take implementing a Singleton class; its so simple that anyone could make one. Most home grown implementations would have race condition though.
1. Obviously, its faster. I, personally, get paid for results, not for how "clever" I am. (Which is good, 'cause I'm not very clever.)
2. Frequently, the answer on the internet is better/more complete/tested. Take implementing a Singleton class; its so simple that anyone could make one. Most home grown implementations would have race condition though.
> Are they a great candidate that cut corners, or a less experienced candidate that used stackoverflow + 10 extra hours to get to a final result.
One of the ideas that is always brought out about the awful interviewing process is that there are just a bunch of totally incompetent programmers out there that need to be kept out of because of fear of bringing quality precipitously down. I'm not sure how many this is and not just bad interviewees based on experiences, but I do also accept that they exist (My guess it's 10-20% of programmers).
At one project I was on we had a guy who was pretty bad (he was then hired away by a competitor to their super programmers team so whatever) but I know he spent days on something that should maybe have taken 1 day, and it looked pretty bad (I mean just looking at it)
I can say that no amount of time would have allowed him to write something that did not look messed up (based on his habit of using map to update multiple arrays at once and not using the returned array from map anywhere)
Maybe they've used extra hours to get a good result, but if they are able to recognize a good result they are not in that small group of programmers that must be kept out at all costs.
One of the ideas that is always brought out about the awful interviewing process is that there are just a bunch of totally incompetent programmers out there that need to be kept out of because of fear of bringing quality precipitously down. I'm not sure how many this is and not just bad interviewees based on experiences, but I do also accept that they exist (My guess it's 10-20% of programmers).
At one project I was on we had a guy who was pretty bad (he was then hired away by a competitor to their super programmers team so whatever) but I know he spent days on something that should maybe have taken 1 day, and it looked pretty bad (I mean just looking at it)
I can say that no amount of time would have allowed him to write something that did not look messed up (based on his habit of using map to update multiple arrays at once and not using the returned array from map anywhere)
Maybe they've used extra hours to get a good result, but if they are able to recognize a good result they are not in that small group of programmers that must be kept out at all costs.
> a bunch of totally incompetent programmers out there that need to be kept out of because of fear of bringing quality precipitously down
I suppose it depends how you define "quality" - in my recent experience, it's always been the "rockstars" that are the worst developers. Because they generally write code which ends up being a nightmare to support and maintain when they're gone. Sure, it's great you wrote your own ORM in Perl but 10 years later it'll be crippling the company because no-one can make any changes without it breaking 27 other things.
(Ok, it's often Perl people...)
For me, a quality developer is someone who achieves the goals of the company in understandable, maintainable, testable, and documented code.
I suppose it depends how you define "quality" - in my recent experience, it's always been the "rockstars" that are the worst developers. Because they generally write code which ends up being a nightmare to support and maintain when they're gone. Sure, it's great you wrote your own ORM in Perl but 10 years later it'll be crippling the company because no-one can make any changes without it breaking 27 other things.
(Ok, it's often Perl people...)
For me, a quality developer is someone who achieves the goals of the company in understandable, maintainable, testable, and documented code.
> For me, a quality developer is someone who achieves the goals of the company in understandable, maintainable, testable, and documented code.
Agree. But it should be noted that the needs of a startup may not always make that easy. I'm personally responsible for transmogrifying an existing Perl ORM for a company, because it only supported a single database handle / class. Which is a pain when you're in a master / slave environment. This hack allowed the company to meet the performance challenges that came along each year by summer. It also allowed growing the company from ~35 people in 2004 to about 5000 people in 2012 (when I left).
It also froze the version of the ORM in question, until it got rewritten from scratch at great expense. But that expense could now be easily carried by the company.
Agree. But it should be noted that the needs of a startup may not always make that easy. I'm personally responsible for transmogrifying an existing Perl ORM for a company, because it only supported a single database handle / class. Which is a pain when you're in a master / slave environment. This hack allowed the company to meet the performance challenges that came along each year by summer. It also allowed growing the company from ~35 people in 2004 to about 5000 people in 2012 (when I left).
It also froze the version of the ORM in question, until it got rewritten from scratch at great expense. But that expense could now be easily carried by the company.
Ok I've had one experience with a guy that I guess thought he was a rockstar, in my opinion he was my level (from looking at the code, although he was much better than me at DevOps), but he did have one awful failing, he didn't want to figure out why something was the way it was and just decided to rewrite everything.
Also he thought I had done something using ElasticSearch because I was an ElasticSearch guy, instead of having learned ElasticSearch because I thought it was what was needed (and no amount of explaining this simple concept could penetrate)
Also he thought I had done something using ElasticSearch because I was an ElasticSearch guy, instead of having learned ElasticSearch because I thought it was what was needed (and no amount of explaining this simple concept could penetrate)
10-20% seems really low in my experience. Do you mainly hire out of the industry?
Hiring heavily from colleges results in 60%+ of applicants that somehow have a B.Sc. from a university that produces amazing applicants with great (3.5+) GPA's but stall at either fizzbuzz or (just in case they memorized fizzbuzz) i-can't-believe-it's-not-fizzbuzz.
Furthermore, some candidates (20ish percent?) can't even answer basic, non-trivia questions about their language they self-report to know best. e.g. Java programmers who either don't know or cannot articulate what the static keyword does.
I'm convinced there's a path through every university, no matter how good, where you can avoid all the good professors and only pick babysitter professors and squeak through your four years with a piece of paper and a well-tuned sensor for which profs actually grade homework instead of just giving 100's so people don't complain.
Hiring heavily from colleges results in 60%+ of applicants that somehow have a B.Sc. from a university that produces amazing applicants with great (3.5+) GPA's but stall at either fizzbuzz or (just in case they memorized fizzbuzz) i-can't-believe-it's-not-fizzbuzz.
Furthermore, some candidates (20ish percent?) can't even answer basic, non-trivia questions about their language they self-report to know best. e.g. Java programmers who either don't know or cannot articulate what the static keyword does.
I'm convinced there's a path through every university, no matter how good, where you can avoid all the good professors and only pick babysitter professors and squeak through your four years with a piece of paper and a well-tuned sensor for which profs actually grade homework instead of just giving 100's so people don't complain.
My experience is only as a programmer, maybe HR has kept most of the bad ones out, but I doubt it based on my experiences with HR. So 10-20% is my experience of how many programmers I encounter are bad in a big organization, and bad is a relative thing of course.
But often when I am brought into a big organization there is one out of every 5 programmers that just should be something else. They're not necessarily stupid, just shouldn't program.
on edit: in my experience small organizations don't have quite as bad because for the organization to be able to exist their programmers must be able to program. (for any organization heavily focused on tech of course)
But often when I am brought into a big organization there is one out of every 5 programmers that just should be something else. They're not necessarily stupid, just shouldn't program.
on edit: in my experience small organizations don't have quite as bad because for the organization to be able to exist their programmers must be able to program. (for any organization heavily focused on tech of course)
I would say that while only 10-20% of programmers might be literally not worth paying any amount for, it might be as much as half of the field that isn’t worth the current, major US market compensation.
once you're willing to admit that they are worth paying something for you're just arguing price, which is whatever the market says it is.
either that or 25 dollars, same as in town.
either that or 25 dollars, same as in town.
A technique that I learned from casting actors is: you don't have to be the candidates enemy to find out if they fit. If you ever find yourself trying to actively make the candidates life harder for no reason, you are doing it wrong (e.g. asking trick questions, expecting weirdly specific answers, etc)
Instead give them multiple different hard (but realistic for the position) problems and be on their side every time to guide them. Try to get them to do the best they can and observe them all the time.
This way you can get good feeling for how they are as a person, how they react to guidance and teamwork, and false negatives where a simple miscommunication leads to wrong solutions and a bad result, despite the candidate actually beeing good, are reduced.
If they constantly and heavily rely on your guidance that is a bad sign. If they actively fight against your guidance, this is bad as well. But in the end you are ”casting” them for a certain position within an existing "cast", and sometimes a person who initially needs more guidance will do great after a while if they are in an environment where they can receive it, while people who actively avoid guidance could work just fine in a position where they don't get too much anyways etc.
The opposite of this approach would be to ask an actor which pieces of [insert obscure european postramatic theatre director here] they know and if they know none assuming they can't act. Bonus points if the project doesn't have any connection to the topic asked at all. If anybody ever does this, it says more about the person asking, than the person answering. No — try really hard to get the best out of the candidates in the most accurate emulation of the job you want to give to them, and have a grasp of what they will get up to speed with in no time and what actually will get worse (e.g. personality traits). And if you areninsecure about someone give it another shot in a second round.
Instead give them multiple different hard (but realistic for the position) problems and be on their side every time to guide them. Try to get them to do the best they can and observe them all the time.
This way you can get good feeling for how they are as a person, how they react to guidance and teamwork, and false negatives where a simple miscommunication leads to wrong solutions and a bad result, despite the candidate actually beeing good, are reduced.
If they constantly and heavily rely on your guidance that is a bad sign. If they actively fight against your guidance, this is bad as well. But in the end you are ”casting” them for a certain position within an existing "cast", and sometimes a person who initially needs more guidance will do great after a while if they are in an environment where they can receive it, while people who actively avoid guidance could work just fine in a position where they don't get too much anyways etc.
The opposite of this approach would be to ask an actor which pieces of [insert obscure european postramatic theatre director here] they know and if they know none assuming they can't act. Bonus points if the project doesn't have any connection to the topic asked at all. If anybody ever does this, it says more about the person asking, than the person answering. No — try really hard to get the best out of the candidates in the most accurate emulation of the job you want to give to them, and have a grasp of what they will get up to speed with in no time and what actually will get worse (e.g. personality traits). And if you areninsecure about someone give it another shot in a second round.
You know, I screwed up an interview that was using your technique once.
The person gave me a piece of paper with a program on it with a number of problems, and asked me to find them. But she was too involved with my problem solving, sort of looking at me as I worked through it, and I was sort of off-balance.
I was almost about to say wait, let me do this on my own, but it was hard for me to be sort of rudely assertive because it was an interview.
Meanwhile other interviews where the person would give me a problem and ignore me would have me doing fine.
The person gave me a piece of paper with a program on it with a number of problems, and asked me to find them. But she was too involved with my problem solving, sort of looking at me as I worked through it, and I was sort of off-balance.
I was almost about to say wait, let me do this on my own, but it was hard for me to be sort of rudely assertive because it was an interview.
Meanwhile other interviews where the person would give me a problem and ignore me would have me doing fine.
Uh that sounds bad. But it wasn't at all what I meant — with actors in a casting you do let them play — but when they are done you either give them a hint and repeat it or you move on to a new thing and tell them what they should focus on.
What you should not do is constantly interrupt while they are doing their thing. Of course you watch them. Beeing watched is part of their job — it isn't for a programmer or someone in tech.
However it can also be hard for actors to feel watched. Contrary to popular expectation they are often not extroverts — this is why it is key to develope some sense of trust beforehand, e.g. I tell them that I will guide them and do all I can to get the best result from them. So I usually make it about solving the problem and not about them.
What you should not do is constantly interrupt while they are doing their thing. Of course you watch them. Beeing watched is part of their job — it isn't for a programmer or someone in tech.
However it can also be hard for actors to feel watched. Contrary to popular expectation they are often not extroverts — this is why it is key to develope some sense of trust beforehand, e.g. I tell them that I will guide them and do all I can to get the best result from them. So I usually make it about solving the problem and not about them.
I understand your point and what you're saying and I like it.
You want a wholistic interview process that treats the interviewers with respect and mimics what the actual job will be like, not some artificial interview scenario.
It was just sort of an anomalous interview for me. I remember being asked, are you sure there's nothing more wrong with the program? and I looked back and couldn't see anything. And she pointed with her pencil and said "how about this?" and I was "Oh, yeah I see that now".
After the interview, of course I was kicking myself.
A friend of mine interviewed for the police, and he told me one of their questions. It was something along the lines of "You pull over a person for drunk driving, and when you get to the car, it's your mother. What do you do?"
My friend said "You call another car to take your mother home"
and I was thinking "what??? that seems so..."
and he said, "Look, they don't want someone on the force who will throw their mother in jail."
sometimes the answer to a novel situation - during a test - only dawns on you later.
You want a wholistic interview process that treats the interviewers with respect and mimics what the actual job will be like, not some artificial interview scenario.
It was just sort of an anomalous interview for me. I remember being asked, are you sure there's nothing more wrong with the program? and I looked back and couldn't see anything. And she pointed with her pencil and said "how about this?" and I was "Oh, yeah I see that now".
After the interview, of course I was kicking myself.
A friend of mine interviewed for the police, and he told me one of their questions. It was something along the lines of "You pull over a person for drunk driving, and when you get to the car, it's your mother. What do you do?"
My friend said "You call another car to take your mother home"
and I was thinking "what??? that seems so..."
and he said, "Look, they don't want someone on the force who will throw their mother in jail."
sometimes the answer to a novel situation - during a test - only dawns on you later.
I think you’ve got it right. The best interviews I’ve participated in felt like working with someone for an hour or so.
As for the opposite scenario, you’ve got that right as well, except that the interviewer is a 25 year old, recently graduated theater major.
As for the opposite scenario, you’ve got that right as well, except that the interviewer is a 25 year old, recently graduated theater major.
> As for the opposite scenario, you’ve got that right as well, except that the interviewer is a 25 year old, recently graduated theater major.
Insert there what you want, hehe. It always reminds me of one teacher I had, who always wanted to get oddly specific answers on his (always a little to unspecific) questions. He was so happy when nobody could answer it, he probably forgot what his job actually was. Behaviour like this tells more about the psyche of the interviewer than it tells about the candidate. Sometimes a behaviour like this is also the interviewers way of making sure the candidate doesn't expose them as the fraud they deep down feel they are: "See they didn't know that oddly specific bit (that I know!), so they must be worse than me – thank god!" In case of my teacher he felt insecure enough to feel the need to one-up teenagers and instead of teaching them something.
Interviewers are only human after all as well. But it shouldn't be the candidates job to rub interviewers in the right way in order to make them comfortable with their own shortcomings as it shouldn't be the pupils job to massage teachers into actually teaching them something.
Insert there what you want, hehe. It always reminds me of one teacher I had, who always wanted to get oddly specific answers on his (always a little to unspecific) questions. He was so happy when nobody could answer it, he probably forgot what his job actually was. Behaviour like this tells more about the psyche of the interviewer than it tells about the candidate. Sometimes a behaviour like this is also the interviewers way of making sure the candidate doesn't expose them as the fraud they deep down feel they are: "See they didn't know that oddly specific bit (that I know!), so they must be worse than me – thank god!" In case of my teacher he felt insecure enough to feel the need to one-up teenagers and instead of teaching them something.
Interviewers are only human after all as well. But it shouldn't be the candidates job to rub interviewers in the right way in order to make them comfortable with their own shortcomings as it shouldn't be the pupils job to massage teachers into actually teaching them something.
My favorite interview process was with Wikimedia. It was a take-home coding project, which I then had to discuss with multiple engineers on the team over the course of four interviews (it was a remote position). I didn't get the job, but it was very enjoyable being able to work on something without someone watching me, and then being able to discuss my design decisions with different engineers who had different opinions about how I could improve my work. It was clearly valuable regardless if I got the position.
When presented with white-boarding interviews, I'll usually feel physically sick. This includes intense sweating and weirdly enough, gassy-ness. I found this old blogpost pretty relatable: https://aneccodeal.blogspot.com/2014/02/interviewing-for-anx...
When presented with white-boarding interviews, I'll usually feel physically sick. This includes intense sweating and weirdly enough, gassy-ness. I found this old blogpost pretty relatable: https://aneccodeal.blogspot.com/2014/02/interviewing-for-anx...
Agreed. I prefer testing how well a candidate can dive into other peoples' code, understand it, and make changes to meet a spec. To respect their at-home and in-person time, that means sending them a small and straightforward codebase, making them extend it in clearly pass-fail ways, and then when they come in you have them talk about the code and extend it a bit more.
The take-home portion serves a few purposes: it filters out uninterested applicants, weeds out non-coders, and lets people prepare in their own time and way (rather than on the spot in an interview). The in-person portion then checks that the candidate can do the work themselves, can communicate about their work, isn't obviously a pain to work with.
Most take-homes test the wrong things: whether someone can start a greenfield project, think up interesting new features, or put in more extra hours than their competition. And most in-person interviews test the wrong things too: have you studied interview problems, are you good at coding and problem-solving out loud under time pressure, and how well do you scrawl syntactically correct code on a whiteboard.
The take-home portion serves a few purposes: it filters out uninterested applicants, weeds out non-coders, and lets people prepare in their own time and way (rather than on the spot in an interview). The in-person portion then checks that the candidate can do the work themselves, can communicate about their work, isn't obviously a pain to work with.
Most take-homes test the wrong things: whether someone can start a greenfield project, think up interesting new features, or put in more extra hours than their competition. And most in-person interviews test the wrong things too: have you studied interview problems, are you good at coding and problem-solving out loud under time pressure, and how well do you scrawl syntactically correct code on a whiteboard.
I have only one thing to add. Don't make it a pre-screen test. I want to know before I do any take-home test that the job is actually a good fit for me.
Wow, I envy you! I wish I could express my disappointment by filling a conference room with noxious gas whenever an interviewer puts me in front of a whiteboard. That is seriously like some kind of superpower.
New graduate here who has this admittedly been on only one side of the interview process. In my interviews I got mostly white boarding problems and one take home. The author was mostly correct in discussing the advantages and disadvantages of both (namely the extra-time for take home assignments and archaic algorithms for white boarding) but I feel there are a couple points I can add.
First, I actually believe white boarding questions (at least the automated kind) are assigned more indiscriminately than take home assignments. Many companies just send you online timed hackerrank or leetcode questions as a preliminary filter.
Second, the entire interview process rewards people who study for the interviews rather than those who like to build projects. Unfortunately I believe companies miss out on a lot of qualified candidates, albeit reducing their false-positives from obscure algorithm-type problems. With so many applicants, reducing false-positives ensures quality hires so it works on the company’s end.
Overall, my opinion is the interview process is relatively broken. Take-home assignments offer more insight into actual development skill, while whiteboard questions filter out a large portion of people (many who would be just as good, if not better, at actual implementations.) But it works for the interviewers so change seems unlikely. This is just my experience for new graduate interviews; perhaps it is different at higher levels.
First, I actually believe white boarding questions (at least the automated kind) are assigned more indiscriminately than take home assignments. Many companies just send you online timed hackerrank or leetcode questions as a preliminary filter.
Second, the entire interview process rewards people who study for the interviews rather than those who like to build projects. Unfortunately I believe companies miss out on a lot of qualified candidates, albeit reducing their false-positives from obscure algorithm-type problems. With so many applicants, reducing false-positives ensures quality hires so it works on the company’s end.
Overall, my opinion is the interview process is relatively broken. Take-home assignments offer more insight into actual development skill, while whiteboard questions filter out a large portion of people (many who would be just as good, if not better, at actual implementations.) But it works for the interviewers so change seems unlikely. This is just my experience for new graduate interviews; perhaps it is different at higher levels.
It could reward those people because it exemplifies following a spec and completing to spec. The whole thing, studying and showing you will do things you don’t want to do (and do them well), and complete a really absurd set of requirements that have seemingly no value.
However, you will stand out if you ask “why did you choose those questions?” Or “how did you come up with those problems?”
Even the best interview that fits in all the check boxes trumps a similar candidate who shows they give a shit.
However, you will stand out if you ask “why did you choose those questions?” Or “how did you come up with those problems?”
Even the best interview that fits in all the check boxes trumps a similar candidate who shows they give a shit.
Wouldn't this be fixed by presenting a on-site assignments? For example a small project, with test suites, where a feature or two needs to be completed and tests updated?
(Disclosure: We did this at a company I worked at).
(Disclosure: We did this at a company I worked at).
> Second, the entire interview process rewards people who study for the interviews rather than those who like to build projects. Unfortunately I believe companies miss out on a lot of qualified candidates, albeit reducing their false-positives from obscure algorithm-type problems. With so many applicants, reducing false-positives ensures quality hires so it works on the company’s end.
There's still quite a lot of signal for an interviewer in someone being ready to do the extra work to achieve the goal (e.g. study for interview) vs. someone how doesn't want to invest time into studying and just wants to throw stuff together. This is especially important for junior developers (since as a company you'll want to train them up long-term to be promoted into higher level), but still remains important for senior developers (since being stuck with someone who refuses to learn new skills is a long-term problem for the company).
There's still quite a lot of signal for an interviewer in someone being ready to do the extra work to achieve the goal (e.g. study for interview) vs. someone how doesn't want to invest time into studying and just wants to throw stuff together. This is especially important for junior developers (since as a company you'll want to train them up long-term to be promoted into higher level), but still remains important for senior developers (since being stuck with someone who refuses to learn new skills is a long-term problem for the company).
I can't speak for others, but for my "senior" level interviews, I do not study. I am coming into these interviews proclaiming a certain number of years of experience in my domain(s). If I have to study, then either I don't know my alleged domain as well as I have claimed, or the questions being asked are irrelevant to said domain.
> There's still quite a lot of signal for an interviewer in someone being ready to do the extra work to achieve the goal (e.g. study for interview) vs. someone how doesn't want to invest time into studying and just wants to throw stuff together.
There's still quite a lot of signal for an interviewer in someone being ready to do the extra work to achieve the goal (e.g. write a real application) vs. someone how doesn't want to invest time into writing a real application and just wants to play with little algorithms and exercises.
Now if you smell a straw man or false dichotomy (hint: writing software instead of cramming leetcode doesn't mean you're not learning new skills; and vice versa) in this, you're on the right track.
The other thing is, what exactly is this signal? What does it tell the interviewer? They have their prejudice, so the signal tells them what they want to hear.
Maybe the actual signal is "I'm bored and desperate and don't have real skills and can't think of anything creative to do so I'll just cram leetcode until some unfortunate company thinks I'm leet and hires me. Because someone on the internet told me that's how you get a job."
This is why I'm not very keen to hand wave about "signal" if there's no way to give it an unambiguous interpretation. In these hiring discussions, I've seen so many examples of "signal" that can be interpreted any number of ways but someone thought they came up with a clever interviewing method so their interpretation of the signal must be right... right?
If it's a hazing ritual and lots of jumping through hoops, the only signal you have is that a person is willing to go through a hazing ritual and jump through hoops. Don't assume anything else. In particular, don't assume it says anything about the skills or capabilities of those who are not willing to go through it.
There's still quite a lot of signal for an interviewer in someone being ready to do the extra work to achieve the goal (e.g. write a real application) vs. someone how doesn't want to invest time into writing a real application and just wants to play with little algorithms and exercises.
Now if you smell a straw man or false dichotomy (hint: writing software instead of cramming leetcode doesn't mean you're not learning new skills; and vice versa) in this, you're on the right track.
The other thing is, what exactly is this signal? What does it tell the interviewer? They have their prejudice, so the signal tells them what they want to hear.
Maybe the actual signal is "I'm bored and desperate and don't have real skills and can't think of anything creative to do so I'll just cram leetcode until some unfortunate company thinks I'm leet and hires me. Because someone on the internet told me that's how you get a job."
This is why I'm not very keen to hand wave about "signal" if there's no way to give it an unambiguous interpretation. In these hiring discussions, I've seen so many examples of "signal" that can be interpreted any number of ways but someone thought they came up with a clever interviewing method so their interpretation of the signal must be right... right?
If it's a hazing ritual and lots of jumping through hoops, the only signal you have is that a person is willing to go through a hazing ritual and jump through hoops. Don't assume anything else. In particular, don't assume it says anything about the skills or capabilities of those who are not willing to go through it.
I completely don't understand interviewers who like to haze the candidates. When I interview a candidate, I'm rooting for him or her. My employer has already invested time and money in the candidate, I have invested the time to study the resume and prepare relevant questions.
My interviews are pretty challenging for the candidates, but I try to make the atmosphere friendly.
They don't care and feel unappreciated. So here's an opportunity to feel better. Who cares if the company needs someone?
Sadly I've even seen this from a company founder who felt that as the company grew they were no longer the "big kahuna" and felt a need to make up for it by showing off in an interview.
(experiences as a co-interviewer not candidate)
Sadly I've even seen this from a company founder who felt that as the company grew they were no longer the "big kahuna" and felt a need to make up for it by showing off in an interview.
(experiences as a co-interviewer not candidate)
They should channel their showing off into displaying how awesome and sociable and good with candidates they are..
if only. A nd they should be glad to be able to work with such smart and solid folks so everything doesn't rest on their shoulders.
The most fun organizations are where everyone is glad to be working with all those other people and things everyone else is smarter than them (so with a lot of opportunities to learn). Organizations like that really do exist -- I've worked in a bunch, but the larger the organization gets the harder that is to maintain.
But when a founder is doing this it poisons the well right away.
The most fun organizations are where everyone is glad to be working with all those other people and things everyone else is smarter than them (so with a lot of opportunities to learn). Organizations like that really do exist -- I've worked in a bunch, but the larger the organization gets the harder that is to maintain.
But when a founder is doing this it poisons the well right away.
Because they're not interviewing people, they're playing the role of interviewer that they once saw on TV.
The goal of the interview is not to subject someone to a test, it's to become informed about them and their abilities to do the job.
The industry has this weird defensive position where everyone believes there's an army of imposters trying to invade them, and it's become a complete parody of itself.
The goal of the interview is not to subject someone to a test, it's to become informed about them and their abilities to do the job.
The industry has this weird defensive position where everyone believes there's an army of imposters trying to invade them, and it's become a complete parody of itself.
Both hazing and cheerleading are bad.
The value of a sunk cost is zero. Who cares what you invested into the process so far? It's time to make the best judgment you can for the benefit of your organization. If you learn that the previous investment was misspent, great. Refine the process for next time.
You should also be providing a positive candidate experience if possible. They may be a customer, have friends you want to hire, or be in an anti-loop and actually good (but you can't tell, and that's okay, because, well, it happens).
You're there to (in order): 1) make the best judgment for the organization 2) represent the company well for the candidate (hopefully this is a faithful representation, since you can't work long term for the company if you're tricking the candidate)
It sounds like you do this already, since your interviews are challenging and friendly. Hopefully they're discriminating at and below job level to allow you to inform a hiring decision. Friendly almost always makes sense (excepting for threatening, dangerous, illegal, or otherwise red flag interviews).
Some houses cut bad interviews short at a halfway point. We don't. The candidate might show you something later in the day that points to another position or area of strength, and, either way, they should feel like they got a fair shake. It also helps with cross-calibration. I've had two candidates that I would have walked out at other companies (less than 1%) at the halfway point. 2-3 incremental person-hours is almost always worth it, though. I'm glad we put in the effort.
The value of a sunk cost is zero. Who cares what you invested into the process so far? It's time to make the best judgment you can for the benefit of your organization. If you learn that the previous investment was misspent, great. Refine the process for next time.
You should also be providing a positive candidate experience if possible. They may be a customer, have friends you want to hire, or be in an anti-loop and actually good (but you can't tell, and that's okay, because, well, it happens).
You're there to (in order): 1) make the best judgment for the organization 2) represent the company well for the candidate (hopefully this is a faithful representation, since you can't work long term for the company if you're tricking the candidate)
It sounds like you do this already, since your interviews are challenging and friendly. Hopefully they're discriminating at and below job level to allow you to inform a hiring decision. Friendly almost always makes sense (excepting for threatening, dangerous, illegal, or otherwise red flag interviews).
Some houses cut bad interviews short at a halfway point. We don't. The candidate might show you something later in the day that points to another position or area of strength, and, either way, they should feel like they got a fair shake. It also helps with cross-calibration. I've had two candidates that I would have walked out at other companies (less than 1%) at the halfway point. 2-3 incremental person-hours is almost always worth it, though. I'm glad we put in the effort.
Cheerleading certainly is bad, but I think candidates should receive a similar level of guidance to what they will receive in the final job (unless they work in isolation or the bad corporate culture needs to be emulated for aome reasons).
Generally the focus of someone hiring should be on the position they are hiring for, and they should actively ask themselves how the candidate with their own specific character traits and skills would cope in that position.
For this the person conducting the interview needs a profound understanding of the open position, which requires work that is often not done — and this results in lazy trick questions and ultimately gets the wrong people hired. Just like in a scientific measurement the person "measuring" the candidate needs to be aware of their own influence. And while taking of half of the stack of applications, throwing them into the bin and pronouncing that you don't like unlucky people certainly reduces your workload, it also reduces your chances of getting the best candidate.
Generally the focus of someone hiring should be on the position they are hiring for, and they should actively ask themselves how the candidate with their own specific character traits and skills would cope in that position.
For this the person conducting the interview needs a profound understanding of the open position, which requires work that is often not done — and this results in lazy trick questions and ultimately gets the wrong people hired. Just like in a scientific measurement the person "measuring" the candidate needs to be aware of their own influence. And while taking of half of the stack of applications, throwing them into the bin and pronouncing that you don't like unlucky people certainly reduces your workload, it also reduces your chances of getting the best candidate.
> Some houses cut bad interviews short at a halfway point. We don't.
There's pros and cons. For on-site, there's a point where you just don't want to waste time.
When I run an interview, if it's clear early on that a candidate is a no, I'll just move the interview along quickly so I don't hurt feelings.
There's pros and cons. For on-site, there's a point where you just don't want to waste time.
When I run an interview, if it's clear early on that a candidate is a no, I'll just move the interview along quickly so I don't hurt feelings.
We have a take-home and a whiteboarding bit. No tricks to the whiteboarding. Very straightforward brute-force solution that you can improve. Take-home doesn't have code. It's a technical design question that you then discuss with an engineer on an on-site.
So far so good. I sometimes attempt to requalify by adding a second step where I try an alternative mechanism of testing, just to see if someone who did poorly on the whiteboard (which is a coderpad-style thing) would do better in a different mechanism. So far very high correlation. Pity, I'm very social so I'd prefer a conversation and walking through code at a higher level.
So far so good. I sometimes attempt to requalify by adding a second step where I try an alternative mechanism of testing, just to see if someone who did poorly on the whiteboard (which is a coderpad-style thing) would do better in a different mechanism. So far very high correlation. Pity, I'm very social so I'd prefer a conversation and walking through code at a higher level.
> You Can Ask the Candidate to Present Code they Wrote.
I have always wondered why this isn't more common, it makes so much sense to me. When you submit an application you usually submit links to your blog, GitHub, projects etc. When I attend an interview I expect us to talk about those things. This is what the person is supposed to be good at. Instead you are met with a predefined set of puzzle questions.
I think the purpose of the interview is to allow the candidate to show what they are good at and what they are excited about and to see if this fits well with what your company is doing.
I have always wondered why this isn't more common, it makes so much sense to me. When you submit an application you usually submit links to your blog, GitHub, projects etc. When I attend an interview I expect us to talk about those things. This is what the person is supposed to be good at. Instead you are met with a predefined set of puzzle questions.
I think the purpose of the interview is to allow the candidate to show what they are good at and what they are excited about and to see if this fits well with what your company is doing.
When I was young and single, I programmed in my free time.
Now I have a wife and three kids and a house and I like woodworking and flying and fixing up old buildings. I sell all of my programming output to my employer, who doesn't happen to make open source. In my free time, I play with my kids or fix something on a house or build something in my shop or something that doesn't leave a mark on GitHub.
If you are evaluating me for a job and ask me to show you some code I've written recently, I can't do it. I do want you to test me, because I want to work in a shop where everybody can pull their weight - I've worked in shops that don't test candidates, and they end up with some really bad programmers who are sometimes decent sweet talkers. But give me something on a whiteboard, or a take home thing that I can finish in a couple hours after the kids go to bed, or just let me bang on a keyboard in your offices or whatever.
If you ask your candidates to show you some code they wrote, you're selecting for candidates who either do open source in their day job, or don't have hobbies that don't produce code. There's lots of great programmers that meet neither of those criteria.
Now I have a wife and three kids and a house and I like woodworking and flying and fixing up old buildings. I sell all of my programming output to my employer, who doesn't happen to make open source. In my free time, I play with my kids or fix something on a house or build something in my shop or something that doesn't leave a mark on GitHub.
If you are evaluating me for a job and ask me to show you some code I've written recently, I can't do it. I do want you to test me, because I want to work in a shop where everybody can pull their weight - I've worked in shops that don't test candidates, and they end up with some really bad programmers who are sometimes decent sweet talkers. But give me something on a whiteboard, or a take home thing that I can finish in a couple hours after the kids go to bed, or just let me bang on a keyboard in your offices or whatever.
If you ask your candidates to show you some code they wrote, you're selecting for candidates who either do open source in their day job, or don't have hobbies that don't produce code. There's lots of great programmers that meet neither of those criteria.
Fair point! I guess if I can edit my statement a bit I'd phrase it like that: If from my application you can see that I have work that we can talk about and discuss, let's do that, if not then let's do the predefined questions. That's fine.
I've had a couple of situations where I go in an interview and the interviews ask me questions that show to me that they have not even looked at my CV and the cover letter they asked me to write. They were just reading off a list of generic tech interview questions.
To me that feels like if I went to interview at a company and told them I have no idea who they are or what they do, I just know their name and that they are looking for a coder. That's just counterproductive.
But I totally agree with you that there is no single solution. I just think there are different "good" questions for different people, interviewers and interviewees need to adapt to that.
I've had a couple of situations where I go in an interview and the interviews ask me questions that show to me that they have not even looked at my CV and the cover letter they asked me to write. They were just reading off a list of generic tech interview questions.
To me that feels like if I went to interview at a company and told them I have no idea who they are or what they do, I just know their name and that they are looking for a coder. That's just counterproductive.
But I totally agree with you that there is no single solution. I just think there are different "good" questions for different people, interviewers and interviewees need to adapt to that.
Live coding (open internet) had been one of the great tools to help evaluate people. We don't want programmers that necessarily know our programming language already, but you should know how to search for map/filter/sort built in functions.
We also leverage take home assignments, which helps people get brushed up before the live challenge... It also tells us where people are in their skills -- code commented? It self-commenting? Any error handing? Etc. I've also pushed for this take home work to be relatively short, it's unfair to have candidates spend 4+ hours in a project with no promise if an interview. It always driver me nuts when businesses asked for that, so I try to avoid it myself.
All this said, it had dramatically United the quality of our candidates we actually see in an interview, which saves our developers time interviewing someone who can't figure out how to do JavaScript sorting using built in functions, or printing out only of numbers in an array of alphanumeric values.
We also leverage take home assignments, which helps people get brushed up before the live challenge... It also tells us where people are in their skills -- code commented? It self-commenting? Any error handing? Etc. I've also pushed for this take home work to be relatively short, it's unfair to have candidates spend 4+ hours in a project with no promise if an interview. It always driver me nuts when businesses asked for that, so I try to avoid it myself.
All this said, it had dramatically United the quality of our candidates we actually see in an interview, which saves our developers time interviewing someone who can't figure out how to do JavaScript sorting using built in functions, or printing out only of numbers in an array of alphanumeric values.
While the example was bad (too general, a potential for copy&paste from somewhere else), it was certainly not for the reason OP wrote:
> The exercise was supposed to take "a few hours" according to the hiring manager, but I'm extremely rusty with ASP.
Well, if they look for an ASP expert, this self-selection saves time on both sides!
I use take-home assignments with boilerplate code, to reduce the initial time investment and make it easier to compare parts of the projects. (Otherwise, it may be a too open-ended task.)
I try to make sure that the skills I require the skills I need. For example, when I needed a deep learning engineer (for a PyTorch project), I expect fluency in at least one deep learning framework. I made it explicit that instead PyTorch boilerplate then can use any other framework (a few people send in Keras).
Yes, people can Google things (good!). And it some sense it balances more junior people (but willing to put more effort) with people with more experience. The later could be told easily by interview. When I ask questions about their solutions, it is easy to see which parts are done with understanding, which - not.
> The exercise was supposed to take "a few hours" according to the hiring manager, but I'm extremely rusty with ASP.
Well, if they look for an ASP expert, this self-selection saves time on both sides!
I use take-home assignments with boilerplate code, to reduce the initial time investment and make it easier to compare parts of the projects. (Otherwise, it may be a too open-ended task.)
I try to make sure that the skills I require the skills I need. For example, when I needed a deep learning engineer (for a PyTorch project), I expect fluency in at least one deep learning framework. I made it explicit that instead PyTorch boilerplate then can use any other framework (a few people send in Keras).
Yes, people can Google things (good!). And it some sense it balances more junior people (but willing to put more effort) with people with more experience. The later could be told easily by interview. When I ask questions about their solutions, it is easy to see which parts are done with understanding, which - not.
OP here.
The reason why the stack requirement bothered me was that, during the phone screening, I basically told the hiring manager that take-homes are fine, but I walk away from anything that has a laundry list of required frameworks.
They were looking for a very senior developer, not someone who would come in and know their stack on day 1.
In my case, the problem wasn't ASP, it was all the other frameworks and patterns they wanted.
The reason why the stack requirement bothered me was that, during the phone screening, I basically told the hiring manager that take-homes are fine, but I walk away from anything that has a laundry list of required frameworks.
They were looking for a very senior developer, not someone who would come in and know their stack on day 1.
In my case, the problem wasn't ASP, it was all the other frameworks and patterns they wanted.
We have a big app and need people who can come in and get up to speed very quickly. A solution to the interview problem that I've adopted over the last few years, to great success, is to conduct a 2 hour, 2 phase interview.
- The first hour is spent having the candidates work through a problem in a sand boxed environment using the actual code base.
- The second hour is spent recapping the work session, discussing their professional background, and general questions about other things they should know to be successful on our team.
The coding test is designed to be solvable within the allotted time and without explicit knowledge of the business rules in the app and only requires an understanding of the actual technology they purport to have the required experience in. Running this test is a great way for us to understand what the candidate's skill level is and what problem solving approaches they use in a real-world scenario. They are free to talk about what they are doing (but is not required) and of course we offer guidance if they get stuck on something. Solving the test is NOT a pre-requisite for getting hired.
In order to reduce anxiety for the candidate, I like to frame the test by telling them to imagine themselves as a short-order contractor who is coming in to help us with a problem that we are having a hard time solving ourselves. I find that this really helps.
The feedback from the candidates for this testing style has always been positive, and I can tell you from post-hire experience, that it has helped us to bring some great developers on board. So far, there have been no false positives. In fact, it's so popular that other teams have either asked me to conduct their interviews or plan on adopting the same methodology for conducting theirs.
If your struggling to get more consistency from your hiring approach, it might be worth a try.
- The first hour is spent having the candidates work through a problem in a sand boxed environment using the actual code base.
- The second hour is spent recapping the work session, discussing their professional background, and general questions about other things they should know to be successful on our team.
The coding test is designed to be solvable within the allotted time and without explicit knowledge of the business rules in the app and only requires an understanding of the actual technology they purport to have the required experience in. Running this test is a great way for us to understand what the candidate's skill level is and what problem solving approaches they use in a real-world scenario. They are free to talk about what they are doing (but is not required) and of course we offer guidance if they get stuck on something. Solving the test is NOT a pre-requisite for getting hired.
In order to reduce anxiety for the candidate, I like to frame the test by telling them to imagine themselves as a short-order contractor who is coming in to help us with a problem that we are having a hard time solving ourselves. I find that this really helps.
The feedback from the candidates for this testing style has always been positive, and I can tell you from post-hire experience, that it has helped us to bring some great developers on board. So far, there have been no false positives. In fact, it's so popular that other teams have either asked me to conduct their interviews or plan on adopting the same methodology for conducting theirs.
If your struggling to get more consistency from your hiring approach, it might be worth a try.
I think it would be interesting to ask a candidate what question they normally ask when they conduct interviews, and have them to walk through the problem. Has anyone here tried that or heard of that being done?
I think you could potentially learn a lot more from that than you could with your own problem. First, there would be a lot less stress on them since it would be a problem they're presumably very familiar with. If they do a terrible job coding it up, well, that's a super clear signal. Second, stuff like how well the do setting up the problem, what kind of problem they choose (is it some generic Leetcode problem, or something they came up with themselves), what they look for in a good solution (just that it works/that it's over-optimized/obeys some arbitrary style standard/etc.) would all provide a less clear signal, but still useful information, especially regarding seniority.
I think you could potentially learn a lot more from that than you could with your own problem. First, there would be a lot less stress on them since it would be a problem they're presumably very familiar with. If they do a terrible job coding it up, well, that's a super clear signal. Second, stuff like how well the do setting up the problem, what kind of problem they choose (is it some generic Leetcode problem, or something they came up with themselves), what they look for in a good solution (just that it works/that it's over-optimized/obeys some arbitrary style standard/etc.) would all provide a less clear signal, but still useful information, especially regarding seniority.
That's such a loaded question, though. It is completely open ended. Will they be judged on the difficulty of the question? Their ability to answer it? Is it even emphasizing the right area for this interview?
I strongly believe the candidate should be given every possible direction on what they are being judged on.
I'll ask, "What are some of the things you look for in a code review?" If they hesitate or seem unsure about how to answer, I'll help them out with "What are some of the things that will get a PR rejected?" From there I try to open it up to a conversation rather than getting them to list the "correct" five things to cover in a code review.
I strongly believe the candidate should be given every possible direction on what they are being judged on.
I'll ask, "What are some of the things you look for in a code review?" If they hesitate or seem unsure about how to answer, I'll help them out with "What are some of the things that will get a PR rejected?" From there I try to open it up to a conversation rather than getting them to list the "correct" five things to cover in a code review.
I couldn’t technically answer that question. My company uses a structured interview process, and all our questions fall under NDA.
I've said this before, but here I go again:
The point of an interview is to bring the best out in a Candidate, not to conform that the interviewer knows better, and definitely not to feed u to their ego.
The point of an interview is to bring the best out in a Candidate, not to conform that the interviewer knows better, and definitely not to feed u to their ego.
I might get lot of flak for this, but everytime I see a post criticising the software engineering interview process I'm thinking what is this dude trying to sell.
The interview process is an assessment on your programming knowledge and its also an assessment on whether you'd be able to hang mentally (intellectually) around the people within the organization (read culture).
If the culture of the organization is made of people who are very proficient in computer science theory. It is imperative they'd would expect any new individual to at least be like them. That's human nature!
The interview process is an assessment on your programming knowledge and its also an assessment on whether you'd be able to hang mentally (intellectually) around the people within the organization (read culture).
If the culture of the organization is made of people who are very proficient in computer science theory. It is imperative they'd would expect any new individual to at least be like them. That's human nature!
That's exactly what I'm trying to sell! You make my point better than me!
I'm also not criticising the process... The process is good! "The problem is bad interviews" is meant to direct criticism to bad interviews.
A religious war of whiteboard vs take-home vs whatever interview format is in fashion is silly.
I'm also not criticising the process... The process is good! "The problem is bad interviews" is meant to direct criticism to bad interviews.
A religious war of whiteboard vs take-home vs whatever interview format is in fashion is silly.
Is hiring broken for non-software jobs like chemists, physicists? What about writers or journalists?
I only have vague insights into these industries but science jobs afaik rely a lot on recommendations - on calling prior collegues and asking. And for writers - well reading their previous stuff works well (easier with open source stuff)
So i would suggest right approach is more a simple FizzBuzz screen, a cultural fit (careful of biases!) and then reading their past OSS / other work (so maybe a prepared portfolio is a good "take home") and calling their references.
I only have vague insights into these industries but science jobs afaik rely a lot on recommendations - on calling prior collegues and asking. And for writers - well reading their previous stuff works well (easier with open source stuff)
So i would suggest right approach is more a simple FizzBuzz screen, a cultural fit (careful of biases!) and then reading their past OSS / other work (so maybe a prepared portfolio is a good "take home") and calling their references.
then reading their past OSS / other work
A lot of people don't have any polished open source code. I have a lot of stuff on Github, but I wrote it for myself. I don't care if it brakes If I only using it once or twice a year.
A lot of people don't have any polished open source code. I have a lot of stuff on Github, but I wrote it for myself. I don't care if it brakes If I only using it once or twice a year.
From the point of view of interviewer.
I find that take-home exercises have two major flaws:
1. The push the effort onto interviewee. Interviewee should not be punished for the interviewer's inability to prepare good questions and exercises and inability to design the process so that bad candidates are filtered out early in the process.
2. They are not really meaningful. I've been there as a candidate. It is difficult to voluntarily stick to the timelines given when you know you could do this or do that to improve the solution and that other candidates will not hesitate to do it. As an interviewer, I have no idea if the candidate did the exercise by themselves or with outside help. Some (not all) candidates will do anything to game the system. You don't want to get these people onboard.
Based on that I am strongly against take home tasks.
On the other hand I like whiteboard exercises because they let me discuss things in an environment and closely to circumstances we could be having at work.
It is normal for me to draw algorithms or even implement pieces of (pseudo) code on the whiteboard and to use it for discussion.
What I try to keep in mind that different people deal differently with the whiteboard. Some people like to visualize, some think a little bit more abstract. I had to work hard to be able to draw anything in a way that will be digestible by other people and I still think am doing only mediocre job of it.
I find that take-home exercises have two major flaws:
1. The push the effort onto interviewee. Interviewee should not be punished for the interviewer's inability to prepare good questions and exercises and inability to design the process so that bad candidates are filtered out early in the process.
2. They are not really meaningful. I've been there as a candidate. It is difficult to voluntarily stick to the timelines given when you know you could do this or do that to improve the solution and that other candidates will not hesitate to do it. As an interviewer, I have no idea if the candidate did the exercise by themselves or with outside help. Some (not all) candidates will do anything to game the system. You don't want to get these people onboard.
Based on that I am strongly against take home tasks.
On the other hand I like whiteboard exercises because they let me discuss things in an environment and closely to circumstances we could be having at work.
It is normal for me to draw algorithms or even implement pieces of (pseudo) code on the whiteboard and to use it for discussion.
What I try to keep in mind that different people deal differently with the whiteboard. Some people like to visualize, some think a little bit more abstract. I had to work hard to be able to draw anything in a way that will be digestible by other people and I still think am doing only mediocre job of it.
I’ve always wondered how mechanical or civil engineers are interviewed? How about accountants? You never hear about interview problems for them.
As an industry we should take our dignity back.
As an industry we should take our dignity back.
Those professions usually require a license gated by a standardized exam, which ensures some level of competency before showing up to the interview.
One thing I prefer about whiteboard coding interview exams is that the do require an investment of time from the interviewing company. Take-homes allow them to externalize the horrendous inefficiency of the interviewing process.
Beyond that, I think articles like this are important to read and circulate widely. Young people need to know that if they become doctors, lawyers, nurses, actuaries, licensed processional engineers, dentists, or financial planners, they will indeed need to take a test. However, the test will be managed by a central authority, with predictable content, with a clear study path, and once you take the entrance exam, you won't have to take it over and over during job interviews for the rest of your life. There may be refresher exams, but again, they will be managed by the licensing body and will provide a clear preparation path, and will be graded consistently. It won't all be left to the whims of whoever thought up a question.
If you become a dev, you will deal with interview exams, often conducted by people who are not really qualified to conduct exams, every time you interview, even if you have decades of experience. If you are a senior actuary, you will not need to integrate by parts every time you interview, but if you are a developer, you may very well have to find all matching subtrees in a binary tree every time you interview. You are signing up for a lifetime of retaking your intro to algorithms midterm.
You may be able to avoid this by starting a consulting company, or going into management, but at this point, if you plan to be a "knowledge worker", I would highly recommend taking a look at other fields.
Beyond that, I think articles like this are important to read and circulate widely. Young people need to know that if they become doctors, lawyers, nurses, actuaries, licensed processional engineers, dentists, or financial planners, they will indeed need to take a test. However, the test will be managed by a central authority, with predictable content, with a clear study path, and once you take the entrance exam, you won't have to take it over and over during job interviews for the rest of your life. There may be refresher exams, but again, they will be managed by the licensing body and will provide a clear preparation path, and will be graded consistently. It won't all be left to the whims of whoever thought up a question.
If you become a dev, you will deal with interview exams, often conducted by people who are not really qualified to conduct exams, every time you interview, even if you have decades of experience. If you are a senior actuary, you will not need to integrate by parts every time you interview, but if you are a developer, you may very well have to find all matching subtrees in a binary tree every time you interview. You are signing up for a lifetime of retaking your intro to algorithms midterm.
You may be able to avoid this by starting a consulting company, or going into management, but at this point, if you plan to be a "knowledge worker", I would highly recommend taking a look at other fields.
Why don't they trust the universities?
If the candidate has a degree in computer science shouldn't he be able to code ?
OP here:
Universities are not trade schools, they are often focused on education for the sake of pursuing knowledge, not how to train you for everything you need in your career.
Computer science is not software engineering; think of the difference between physics and mechanical engineering.
A lot of computer science classes are highly theoretical and have only tangential relationship to what it takes to write a commercially successful program.
BTW: This is the similar for doctors, in the US, your doctor has to get a license in addition to a degree, and the license requires self-study in topics that aren't covered in medical school.
Universities are not trade schools, they are often focused on education for the sake of pursuing knowledge, not how to train you for everything you need in your career.
Computer science is not software engineering; think of the difference between physics and mechanical engineering.
A lot of computer science classes are highly theoretical and have only tangential relationship to what it takes to write a commercially successful program.
BTW: This is the similar for doctors, in the US, your doctor has to get a license in addition to a degree, and the license requires self-study in topics that aren't covered in medical school.
While I think that ought to be a requirement for getting a CS degree, not all universities do. Moreover, being able to code isn't a boolean - while some people are indeed totally incompetent, among the competent people, some are better than others, and some have strengths and weaknesses that would make them more ore less suited to a particular role. It behooves an employer to try to measure these things.
Computer science doesn’t teach you how to write code.
Not in my experience.
The same applies to algorithmic challenges given for whiteboard coding. The fact one can reverse a linked list doesn't mean one would be a good developer in a real world project. Whiteboard coding doesn't check the basic system design skills.
Whiteboard coding isn't designed to find good all-round developers, it's designed to filter the candidate pool to decrease the likelihood of candidates being selected that don't have the necessary coding chops. Other aspects of the selection process check other aspects of the desired skill set.
... No.
Look, if you spend five years with a girl, you’ve gotta marry her. Clearly, you didn’t find anything wrong with her enough to break up and she can’t wait around forever. Are there better women out there? Almost definitely. They didn’t just commit five years to your chubby ass though.
Likewise, once you make someone put 4+ hours into tech work and it’s good work, you’d better hire them or apologize profusely for wasting their time.
Likewise, once you make someone put 4+ hours into tech work and it’s good work, you’d better hire them or apologize profusely for wasting their time.
You should be looking at one woman over those five years.
For companies, you are probably looking at multiple developers and if you do only have one opening then even if more than one is good there is going to be a best.
For companies, you are probably looking at multiple developers and if you do only have one opening then even if more than one is good there is going to be a best.
The company is only going hire one person. If they make ten people do their task, they've committed themselves to wasting the time and energy of all at least nine of those people, even if all ten people do something amazing with that assignment.
One of my favorite interviews ever was squarely in-between whiteboarding and take-home. I was put in a room by myself with access to the internet and a computer and asked to write a specific game in any language I wanted. The rules of the game were given, but everything else was up to me. The main requirements were to be able to make a move, and to calculate the current score. For extra credit, make an AI player.
There were other phases of the interview, coding wasn’t the whole thing, but I really enjoyed it. I bombed the object modeling on a whiteboard part of the interview, but the company made a great offer anyway based on being able to code. I’m modeling my hiring now on this type of interview because it was both fun and effective.
There were other phases of the interview, coding wasn’t the whole thing, but I really enjoyed it. I bombed the object modeling on a whiteboard part of the interview, but the company made a great offer anyway based on being able to code. I’m modeling my hiring now on this type of interview because it was both fun and effective.
I opted for an easy take home from Twitter recently when applying for a job there. However, the problem was so horribly written and vague that I couldn't figure out what they really wanted, even after asking questions. Honestly, I would have been better doing a problem with time constraints interactively, at least I could get some decent feedback about what was expected.
Flubbing the Twitter take home worked out for the best, Google was a much better fit for me anyways. Their interview process was great also, and doing whiteboarding over VC affords certain advantages (getting to type in code, if only into Google docs, is a huge win for me, Jamboard + an iPad Pro worked out great as well).
Flubbing the Twitter take home worked out for the best, Google was a much better fit for me anyways. Their interview process was great also, and doing whiteboarding over VC affords certain advantages (getting to type in code, if only into Google docs, is a huge win for me, Jamboard + an iPad Pro worked out great as well).
When I interviewed at Google last year, they said they offered every candidate a Chromebook on which to type code instead of using a whiteboard, but the office I visited had no working interviewee Chromebooks. Maybe most people who go to on-sites can type code on a real computer.
As an interviewee, I have a particular distaste for take home questions, at least if given before doing a couple phone screens- I want to see a company invest time in me a bit before spending half a day coding.
Besides, having been an interviewer more than an interviewee at this point I find being able to competently discuss previous projects with a degree of excitement more predictive of success anyway. I still ask to see interview coding, but it's more a check box for basic skills than a major focus
Besides, having been an interviewer more than an interviewee at this point I find being able to competently discuss previous projects with a degree of excitement more predictive of success anyway. I still ask to see interview coding, but it's more a check box for basic skills than a major focus
When I'm giving candidate a take-home assignment - I'm always asking for time and effort estimation, both "net" and "gross", like in "how many hours will it take" and "when it will be ready". Actually, ability to give viable estimation and stick to it is more important than coding itself.
As for other matters in article, my current company just paying candidate for finished assignment, plain and simple.
As for other matters in article, my current company just paying candidate for finished assignment, plain and simple.
Back when I used to do whiteboard interviews, the trick was not to grade the candidate on right or wrong answers, but to look for signs of engineering thinking. The problem was simple, but well-suited to low-level languages like C, and we would even give them hints if they forgot some aspect of C-family syntax. BSers would reveal themselves in obvious ways, and anyone who displayed engineering competence was still in consideration.
Algorithms interviews are IQ tests, the end.
This isn't necessarily a bad thing. But it's not only an IQ test: it also assesses how good is someone at solving problems. The individual shouldn't come with the perfect answer in a limited amount of time but show a good thinking process which put him on the right track to solve the problem.
When doing coding someone can have to solve an issue which doesn't have an answer on stackoverflow, so having problem solving skills is valuable.
Also, being good at algorithmics and data structures enable someone to come up with better solutions which might matter in a particular case.
I think algorithmic questions are better than asking questions about random methods from a particular framework.
When doing coding someone can have to solve an issue which doesn't have an answer on stackoverflow, so having problem solving skills is valuable.
Also, being good at algorithmics and data structures enable someone to come up with better solutions which might matter in a particular case.
I think algorithmic questions are better than asking questions about random methods from a particular framework.
It also assesses how good someone is at preparing to solve specific kinds of problems.
Which is fine. If you're naturally talented at whiteboarding, great. If not, but you've prepared months ahead of time, great also.
If you're not naturally talented, and haven't prepared because of naivety or due to some principle, well, that's not exactly a great signal to an employer.
Which is fine. If you're naturally talented at whiteboarding, great. If not, but you've prepared months ahead of time, great also.
If you're not naturally talented, and haven't prepared because of naivety or due to some principle, well, that's not exactly a great signal to an employer.
More like Algorithms interviews are a way for software engineers to stroke their own ego, the end.
If software engineers actually took the time to look at how intelligence is actually measured in fields outside their own, and how it relates to job performance, we'd have a lot less broken interviews.
If software engineers actually took the time to look at how intelligence is actually measured in fields outside their own, and how it relates to job performance, we'd have a lot less broken interviews.
No, it's actually a quite opposite - trivia test from the field of Competitive Programming.
I actually enjoy take-home exams more, because a good whiteboard interview absolutely relies on having a good / competent interviewer.
Take-home exams obviously also need competent people to analyze the results, and also have their pitfalls - especially if they're just being used as free labor.
Take-home exams obviously also need competent people to analyze the results, and also have their pitfalls - especially if they're just being used as free labor.
I really enjoyed this article. It's probably one of the most fair, unbiased, and insightful articles written on tech interviews and may be the best one I've read.
Over the past 15 years, I have interviewed with many different companies. I once read that when you're a good fit for the company, the interview will feel suspiciously easy. This has been the case for all the times I've gotten offers.
Reading this article, a lot of the good points remind me of a particular interview I had though. It was at relatively large tier-2 tech firm a few years ago (not the FAANG tier); one where I didn't get an offer, but the interview process and experience was memorably good. I remember specifically telling the recruiter even after the rejection that I really enjoyed the interview there.
One of the rounds was with the principal engineer at the company, who was the most senior engineer at the company at the time if I recall correctly. It was the system architecture interview. I was asked to prepare by thinking of a system I had designed and architected in my past experience, and present it as if the principal engineer was my coworker and we would hold tech discussions. It's the closest thing I've ever gotten to the "Present code they wrote" idea that Andrew mentions here.
For the coding round of this interview, I was asked to bring my own laptop (another point that Andrew mentioned here; this was also the only company of all the interviews I've had in 15 years that asked me to do this.). I was then given a problem at the beginning of that interview, and given 45 minutes or so to write code on my own (the interviewer actually walked out of the room and left me alone while I did). It was a reasonable problem to code for the allotted time, and had reasonable complexity but not overly difficult. When time's up, the interview came back and we discussed my code.
Both of those rounds were something that was so well done that I think it was the most fair way a company could have assessed me correctly. I did well on the coding round, but my experience in system architecture was not a good match for the type of work they did, and in retrospect I could tell. Since I was interviewing at the Staff Engineer level, I think they correctly evaluated me and turned me down for the role. (Although I would point out that after that day of interview, they ended up being undecided, and asked me to come back one more time to have one more interview round, with one of the first engineers of the company there. I did poorly in that round for a variety of reasons, and that was ultimately what got me the rejection.)
Over the past 15 years, I have interviewed with many different companies. I once read that when you're a good fit for the company, the interview will feel suspiciously easy. This has been the case for all the times I've gotten offers.
Reading this article, a lot of the good points remind me of a particular interview I had though. It was at relatively large tier-2 tech firm a few years ago (not the FAANG tier); one where I didn't get an offer, but the interview process and experience was memorably good. I remember specifically telling the recruiter even after the rejection that I really enjoyed the interview there.
One of the rounds was with the principal engineer at the company, who was the most senior engineer at the company at the time if I recall correctly. It was the system architecture interview. I was asked to prepare by thinking of a system I had designed and architected in my past experience, and present it as if the principal engineer was my coworker and we would hold tech discussions. It's the closest thing I've ever gotten to the "Present code they wrote" idea that Andrew mentions here.
For the coding round of this interview, I was asked to bring my own laptop (another point that Andrew mentioned here; this was also the only company of all the interviews I've had in 15 years that asked me to do this.). I was then given a problem at the beginning of that interview, and given 45 minutes or so to write code on my own (the interviewer actually walked out of the room and left me alone while I did). It was a reasonable problem to code for the allotted time, and had reasonable complexity but not overly difficult. When time's up, the interview came back and we discussed my code.
Both of those rounds were something that was so well done that I think it was the most fair way a company could have assessed me correctly. I did well on the coding round, but my experience in system architecture was not a good match for the type of work they did, and in retrospect I could tell. Since I was interviewing at the Staff Engineer level, I think they correctly evaluated me and turned me down for the role. (Although I would point out that after that day of interview, they ended up being undecided, and asked me to come back one more time to have one more interview round, with one of the first engineers of the company there. I did poorly in that round for a variety of reasons, and that was ultimately what got me the rejection.)
[deleted]
Whiteboard interviews all the way. They are interactive. And they can be two-way interviews. All participants of the interview are interviewing each other.
Take homes are basically free labor.
Take homes are basically free labor.
I feel that we (techies) make hiring miserable by asking for the perfect versions of ourselves.
The result is entry level inflationism.
The result is entry level inflationism.
CS whiteboarding makes sense when tech stack isn't going to be constant and the tasks you might be doing requires more than making calls to backend from MVC. On the other hand, take-home stack specific question makes sense otherwise.
The bottom line is that software engineering jobs are very different but we keep expecting to put on same cloths on them in terms of interview process.
The bottom line is that software engineering jobs are very different but we keep expecting to put on same cloths on them in terms of interview process.
> CS whiteboarding makes sense when tech stack isn't going to be constant and the tasks you might be doing requires more than making calls to backend from MVC.
I'm not inclined to agree. If I look at my job, well, it's rather "full-stack", spanning from hardware to bootloader to kernel to init system to application level. Debugging techniques range from probing with an oscilloscope, disassembly, dissecting core dumps, printf-tracing of asynchronous multithreaded systems, packet captures and traffic analysis at raw ethernet level..
I can't think of a single "CS whiteboarding" question that would do much to help me select a candidate for my position. As far as algo wizardry goes.. dude, naive O(n) and O(n*n) algorithms work for damn well near everything, and if you can figure out how to use one of the existing binary tree libraries or call bsearch(), you're pretty much set. I'm sure we could give you a few days to figure out some more advanced algorithm if profiling proves it to be necessary for some particular job.
Honestly I think the most important skill is the ability to learn new stuff and dig deep. I don't know how to test for that in an interview, I really have no clue.
Probably understanding concurrency and races is one of the more "critical skills" and I'd like to see that in a candidate, but I don't know if lacking a good grasp of that is grounds for rejecting a candidate. All too often I see lists prerequisite "critical skills" that any good programmer would be able to pick up on the job as needed. Do you really need to test people for something they can figure out in a day or three and get good at over the coming few weeks?
I'm not inclined to agree. If I look at my job, well, it's rather "full-stack", spanning from hardware to bootloader to kernel to init system to application level. Debugging techniques range from probing with an oscilloscope, disassembly, dissecting core dumps, printf-tracing of asynchronous multithreaded systems, packet captures and traffic analysis at raw ethernet level..
I can't think of a single "CS whiteboarding" question that would do much to help me select a candidate for my position. As far as algo wizardry goes.. dude, naive O(n) and O(n*n) algorithms work for damn well near everything, and if you can figure out how to use one of the existing binary tree libraries or call bsearch(), you're pretty much set. I'm sure we could give you a few days to figure out some more advanced algorithm if profiling proves it to be necessary for some particular job.
Honestly I think the most important skill is the ability to learn new stuff and dig deep. I don't know how to test for that in an interview, I really have no clue.
Probably understanding concurrency and races is one of the more "critical skills" and I'd like to see that in a candidate, but I don't know if lacking a good grasp of that is grounds for rejecting a candidate. All too often I see lists prerequisite "critical skills" that any good programmer would be able to pick up on the job as needed. Do you really need to test people for something they can figure out in a day or three and get good at over the coming few weeks?
Again, you are assuming all software engineering jobs are like yours. I’d been on projects where I’d to deeply think about graphs and literally invent variations of algorithms that worked for my scenario. I’d to written code for tree searching, online median finding, distributed minimum tree spanning etc in my various projects. If you are working in things search, AR, DL related products anyone will tell you that 1% of improvement in algo performance often translates to millions of dollars of savings. Unfortunately designing and evaluating algorithms is very hard earned skills and candidates who have internalized tools of this trade do much better than regular full stack folks. CS whiteboarding is precisely for such candidates but unfortunately full stack guys also started adopting them for their jobs. For full stack projects that need little to no algorithms design, take home coding question specific to tech stack works much better.
I've been asked to do coding interviews for freelance gigs a couple of times recently. This does not make sense to me. The whole point of freelancing is that either of us can terminate the contract on short notice if for whatever reason we don't like each other. Usually, if it comes up, I just recommend the company that if they have another qualified candidate (i.e. better qualified than me), they should do their best to hire them. It's a good way to cut the bullshit and get it in people's heads that they need to work hard to get me if they want me to commit to their project. If based on my profile they are not super eager to talk to me, I'm wasting my time because either they need somebody with a different profile or they are B's trying to hire C's. In both cases, that's a good reason for me to not engage any further.
I've been in the interviewer role as well and I tend to not rely on coding exercises, whiteboard BS, etc. I usually just wing it knowing full well that interviews are in any case a bit of a crap shoot in terms of objective results. My goals when interviewing are very simple: 1) verify that what is in the person's CV holds up and try to get them to talk about what they did, how they did it, etc. 2) inject a few technical questions to see if they can come up with some good answers or whether they start bullshitting. 3) Figure out what they don't know and how they would deal with having to learn something new. 4) Figure out how well their experience aligns with my team's needs. I'm OK with a less than perfect match if a person has the potential to learn new stuff. Especially with junior roles, that's the most important thing because they probably don't know much yet.
But really, I tend to go with my first impressions here and have learned the hard way that ignoring those leads to bad hires. Yes, there's a risk I'm wrong about not hiring somebody. But if it feels wrong, it's not going to happen. Usually, I know within a few minutes what it's going to be; the rest of the interview is then either about making sure the candidate is sold on the job or giving me enough material to defend the choice not to hire to my managers, HR, etc. Coding exercises don't really factor into this. If I have any doubts whatsoever about a person's ability to actually produce code, I'm not going to hire them.
Unfortunately hiring is hard. There are a lot of mediocre candidates out there with a lot of mediocre recruiters representing them. I vastly prefer warm introductions through trusted friends and colleagues. So, as an interviewer, I deal with screening a lot of obviously not great candidates. The flip side is that when somebody good comes along, you need to be able to act fast because they'll have other options. Coding exercises are an obstacle for both sides.
I've been in the interviewer role as well and I tend to not rely on coding exercises, whiteboard BS, etc. I usually just wing it knowing full well that interviews are in any case a bit of a crap shoot in terms of objective results. My goals when interviewing are very simple: 1) verify that what is in the person's CV holds up and try to get them to talk about what they did, how they did it, etc. 2) inject a few technical questions to see if they can come up with some good answers or whether they start bullshitting. 3) Figure out what they don't know and how they would deal with having to learn something new. 4) Figure out how well their experience aligns with my team's needs. I'm OK with a less than perfect match if a person has the potential to learn new stuff. Especially with junior roles, that's the most important thing because they probably don't know much yet.
But really, I tend to go with my first impressions here and have learned the hard way that ignoring those leads to bad hires. Yes, there's a risk I'm wrong about not hiring somebody. But if it feels wrong, it's not going to happen. Usually, I know within a few minutes what it's going to be; the rest of the interview is then either about making sure the candidate is sold on the job or giving me enough material to defend the choice not to hire to my managers, HR, etc. Coding exercises don't really factor into this. If I have any doubts whatsoever about a person's ability to actually produce code, I'm not going to hire them.
Unfortunately hiring is hard. There are a lot of mediocre candidates out there with a lot of mediocre recruiters representing them. I vastly prefer warm introductions through trusted friends and colleagues. So, as an interviewer, I deal with screening a lot of obviously not great candidates. The flip side is that when somebody good comes along, you need to be able to act fast because they'll have other options. Coding exercises are an obstacle for both sides.
- take home interviews incentivize the candidate to spend more time than they’re told. You can choose a three hour problem, but candidates that really want the job will probably spend more time. Less experienced candidates will spend more time on it too.
- I never found reviewing take home interviews to be a great indicator of skill level. It’s hard to know how long was really spent on it, and how much the candidate used resources etc to guide them. Are they a great candidate that cut corners, or a less experienced candidate that used stackoverflow + 10 extra hours to get to a final result.
- whiteboarding can go badly too. The caveat is that if it does, only so much of the candidate’s time is taken up; there’s a hard ceiling.
- from the perspective of conducting whiteboard interviews, the company loses out if the interviewer doesn’t do a good job. This is really on the company though - it’s worth investing in a good whiteboarding process. If there’s no investment here, it’s only the company that suffers (competent candidates will get hired by a company with a better process).
In terms of conducting a good whiteboard interview, my $0.02 is no trick questions, just solid problem solving with plenty of scope to show creativity and proficiency.