Take-home interviews(blog.triplebyte.com)
blog.triplebyte.com
Take-home interviews
http://blog.triplebyte.com/take-home-interviews
294 comments
After doing a handful of these and rejecting several handful of these tests I'd like to add a little to you comment.
I agree that the time it takes is always muchuch longer than what they state.
Companies that offer these tests before doing an initial phone screen get that email deleted. Why would I as an applicant who is applying to 10+ jobs spend time doing this test when I have never even had a chance to interact with a human.
The tests are sometimes not even close to the actual job. As in the job description is for a front end developer and JavaScript knowledge needed and they ask you to write the test using a completely different language. one company (who adversities jobs on here all the time) asked to do some php command line scripting for a JavaScript front end position. How is that in anyway a useful judgement of someone's skills. So wasting people's time is a big deliminator.
An example of a company I experienced that did the take home 'right' did an initial phone screen a couple days after applying. Then did another tech screen which was just basic stuff. After that they asked me to do a take home exercise and while completing it they continued to move forward with the application process including setting up travel arrangements. The take home test was directly related to the job and was given open ended for some creativity if one chose. The onsite final interview was discussing the code, so it would do little good to cheat on it because you need to be able to talk through it.
I didn't even get the job with them but it was actually not a painful experience for once to do a take home test.
Just my two cents, but I believe there is a good way and a terrible way to do it.
I agree that the time it takes is always muchuch longer than what they state.
Companies that offer these tests before doing an initial phone screen get that email deleted. Why would I as an applicant who is applying to 10+ jobs spend time doing this test when I have never even had a chance to interact with a human.
The tests are sometimes not even close to the actual job. As in the job description is for a front end developer and JavaScript knowledge needed and they ask you to write the test using a completely different language. one company (who adversities jobs on here all the time) asked to do some php command line scripting for a JavaScript front end position. How is that in anyway a useful judgement of someone's skills. So wasting people's time is a big deliminator.
An example of a company I experienced that did the take home 'right' did an initial phone screen a couple days after applying. Then did another tech screen which was just basic stuff. After that they asked me to do a take home exercise and while completing it they continued to move forward with the application process including setting up travel arrangements. The take home test was directly related to the job and was given open ended for some creativity if one chose. The onsite final interview was discussing the code, so it would do little good to cheat on it because you need to be able to talk through it.
I didn't even get the job with them but it was actually not a painful experience for once to do a take home test.
Just my two cents, but I believe there is a good way and a terrible way to do it.
I agree. I've done really good ones and really bad ones. A good test started in the office with the hiring manager. Really them just watching me write a simple CRUD app in asp.net when it was all the rage. This was for an entry level gig and was actually a really good test. At the end they had an additional feature for you to add from home. It took a couple hrs and ended up being a great test to find decent entry level ppl.
The bad one was about 2 yrs ago. I walked into a conference room for the final round, was sat down at a mac, and asked to write some code in their proprietary DSL without any documentation or anything. It was maddening. I almost walked out, but the salary was stupid high. Didn't get the job. Thinking back on it, maybe it was just a test and they did want me to say it was ridiculous.
The bad one was about 2 yrs ago. I walked into a conference room for the final round, was sat down at a mac, and asked to write some code in their proprietary DSL without any documentation or anything. It was maddening. I almost walked out, but the salary was stupid high. Didn't get the job. Thinking back on it, maybe it was just a test and they did want me to say it was ridiculous.
I almost walked out, but the salary was stupid high. Didn't
get the job. Thinking back on it, maybe it was just a test
and they did want me to say it was ridiculous.
- They are being unintentionally stupid by asking you to do this activity.
- If they are intentionally asking you to do a stupid thing and they expect you to revolt in the interview and decry the madness. How does that make you feel about your future day-to-day life at that organization?
In both cases those are not good "A-player" types of people to work for.
No need to torture yourself over more favorable prospects "lost" :)
> Thinking back on it, maybe it was just a test and they did want me to say it was ridiculous.
If that was the case ... then you don't want to work there. I've worked with a few business people that intentionally filtered interviewees by strange characterists. [Aka... graphic designers that were juinor, but didn't have web dev experience (I want to say it was something stranger than that)]
If that was the case ... then you don't want to work there. I've worked with a few business people that intentionally filtered interviewees by strange characterists. [Aka... graphic designers that were juinor, but didn't have web dev experience (I want to say it was something stranger than that)]
[deleted]
One take-home test I declined was for a advertised web job that turned out to require lots of PHP knowledge and plugin editing experience.
They seemed offended when I was frank that it would take much longer than the 1 hour they were quoting and that it wouldn't be a real demonstration of skills but more so what I could cram.
OTOH, my current work does a sort of take-home test but it can't be faked/cheated on because you have to record yourself teaching something.
They seemed offended when I was frank that it would take much longer than the 1 hour they were quoting and that it wouldn't be a real demonstration of skills but more so what I could cram.
OTOH, my current work does a sort of take-home test but it can't be faked/cheated on because you have to record yourself teaching something.
The best take-home interview I ever did had a time-limit. And of course, I panicked when getting close to that limit, and tried my best to tidy up my solution but didn't really manage. Then a pop-up appeared asking "Would you like another 5 minutes?" That was just enough time to calm down, tidy up sufficiently, and submit clean code.
It seemed like a good middle ground to me.
It seemed like a good middle ground to me.
The only time I've been given one there was a time limit as it was fairly strict. Frankly I'd have preferred an untimed version but I can see why people would disagree with that.
"I can see why people would disagree with that."
Yeah, I can see why too, since almost all software is written with a literal ticking time bomb ready to go off if you don't finish it within a certain minute.
This is, of course, sarcasm. Timed programming interview tests are fucking dumb.
I mean, if an employer gives someone a take home test that they expect to take 3 hours and they give them a week to do it, that's reasonable if technically "timed"; but if an employer gives someone a take home test that they expect to take 3 hours and they actually give them 3 hours to do it, that employer sucks at hiring programmers. I won't even bother debating this with the inevitable person who chimes in to rationalize why they do this, because as far as I'm concerned it is a "the sky is blue" statement and any time debating it with people who disagree with this position is completely wasted.
Yeah, I can see why too, since almost all software is written with a literal ticking time bomb ready to go off if you don't finish it within a certain minute.
This is, of course, sarcasm. Timed programming interview tests are fucking dumb.
I mean, if an employer gives someone a take home test that they expect to take 3 hours and they give them a week to do it, that's reasonable if technically "timed"; but if an employer gives someone a take home test that they expect to take 3 hours and they actually give them 3 hours to do it, that employer sucks at hiring programmers. I won't even bother debating this with the inevitable person who chimes in to rationalize why they do this, because as far as I'm concerned it is a "the sky is blue" statement and any time debating it with people who disagree with this position is completely wasted.
One thing you're missing though is that a time limit allows for a controlled variable when you're comparing candidates.
The test I was referring to I'm almost certain was not intended to be completed in the time allowed. I've used it myself for in person interviews (not take home) and give candidates 3x the time I was given, few of them actually finish it completely. While it just made me feel like an abject failure at the time now I realize a big part of what was being tested was given limited time what bits I focused on. In fact that's exactly what I tell candidates who I give it to.
The test I was referring to I'm almost certain was not intended to be completed in the time allowed. I've used it myself for in person interviews (not take home) and give candidates 3x the time I was given, few of them actually finish it completely. While it just made me feel like an abject failure at the time now I realize a big part of what was being tested was given limited time what bits I focused on. In fact that's exactly what I tell candidates who I give it to.
We ask for either work samples or offer the ability to take a take home test. We prefer the former but understand that's not always feasible. If they choose to do the latter we make it clear that it's designed for ~1hr but they can feel free to spend as much or little time as they want. It's not difficult and is really a glorified fizzbuzz, really we're just trying to ascertain if this person can code at all. This is after HR does an initial screen but before one gets into the real engineering group.
I'm a big fan of the work sample method. I have one repo that is polished out and is a good representation of what I am capable of. Most of my code online isn't that nice because I write stuff for fun so making sure I have proper accessibility tags isn't that important.
Now you could debate which is a better representation of my work, the polished dog and pony show repo or the fast and dirty stuff that I push in my ongoing projects.
Now you could debate which is a better representation of my work, the polished dog and pony show repo or the fast and dirty stuff that I push in my ongoing projects.
That topic ("People do stuff at home for an interview") is a topic that comes up again and again. My last reply [1] is about a month old and feels still valid.
Don't state that 'take home tests are the worst'. That is - failing to find better words - crap. If that is the ONLY option, I understand that this might be not for you. But - that's not the case here as far as I can tell. You, as a person interested to interview, can opt in. That is awesome.
Now - you might not be the type of guy that would _want_ to opt in, but please refrain from these absolute statements. No, that's not the worst. In fact, it's probably the _best_ option for a number of people (I myself would - if I'd want to interview with this service - opt for the home project).
I fail to understand how this 'bash the home work' attitude comes up again and again. Yes, don't work for free. But if you're doing a 3h whiteboard marathon or work from your own chair? And you pick which one you prefer? I don't get the hate here..
1: https://news.ycombinator.com/item?id=9770737
Don't state that 'take home tests are the worst'. That is - failing to find better words - crap. If that is the ONLY option, I understand that this might be not for you. But - that's not the case here as far as I can tell. You, as a person interested to interview, can opt in. That is awesome.
Now - you might not be the type of guy that would _want_ to opt in, but please refrain from these absolute statements. No, that's not the worst. In fact, it's probably the _best_ option for a number of people (I myself would - if I'd want to interview with this service - opt for the home project).
I fail to understand how this 'bash the home work' attitude comes up again and again. Yes, don't work for free. But if you're doing a 3h whiteboard marathon or work from your own chair? And you pick which one you prefer? I don't get the hate here..
1: https://news.ycombinator.com/item?id=9770737
Unfortunately, most companies aren't comfortable replacing an onsite with a work sample (or take home.) So, it's just additive.
It's not "the worst" in relation to other interview options. It's the worst because it's usually in addition to other interview options. I just stopped doing take homes on my last round of interviewing. Just not worth it. It was always just added work. It never replaced a stage of the process.
If you think about it, it makes sense though. Very few companies would hire people directly based on the strength of their github account or their topcoder rank. So, if they won't do that, then what extra information does a take home really provide?
Companies seem to recognize that they want to hire people who do good work and that good work isn't done in an interview setting. But very few companies are willing to just analyze the candidate's work. They want to subjectively judge the person.
It's not "the worst" in relation to other interview options. It's the worst because it's usually in addition to other interview options. I just stopped doing take homes on my last round of interviewing. Just not worth it. It was always just added work. It never replaced a stage of the process.
If you think about it, it makes sense though. Very few companies would hire people directly based on the strength of their github account or their topcoder rank. So, if they won't do that, then what extra information does a take home really provide?
Companies seem to recognize that they want to hire people who do good work and that good work isn't done in an interview setting. But very few companies are willing to just analyze the candidate's work. They want to subjectively judge the person.
Agreed (and see the linked comment please): If it's not a replacement for whiteboard bullshit, then GTFO. That's insane and maybe 'the worst'.
But it's important to remember that a take home exercise cannot fully replace an interview. It can replace the coding part, the whiteboard "and now we ask useless trivia questions" part. But there's no way for this work to replace the "would you fit the team" interview.
Others already discuss this in different subthreads here, but basically I'd expect the company to clearly show how their process works and - ideally - filter by ~social~ criterias first ("You might fit the team, if you can code"). Doing work for free with potentially no feedback or a 'fail' in a later discussion is crap, of course.
But it's important to remember that a take home exercise cannot fully replace an interview. It can replace the coding part, the whiteboard "and now we ask useless trivia questions" part. But there's no way for this work to replace the "would you fit the team" interview.
Others already discuss this in different subthreads here, but basically I'd expect the company to clearly show how their process works and - ideally - filter by ~social~ criterias first ("You might fit the team, if you can code"). Doing work for free with potentially no feedback or a 'fail' in a later discussion is crap, of course.
There is a reason that nobody filters by social first.
One of the biggest challenges in hiring is that it takes a ton of time to go through many bad candidates before you see any good ones. And time is the one thing you don't have. You're hiring because you don't have enough resources to do the work you already have.
Therefore the name of the game is efficiently rejecting candidates while using up the least amount of your existing people's time. Which means that the most expensive filters should be done last. And the most expensive one is social because judging it takes time from EVERYONE.
One of the biggest challenges in hiring is that it takes a ton of time to go through many bad candidates before you see any good ones. And time is the one thing you don't have. You're hiring because you don't have enough resources to do the work you already have.
Therefore the name of the game is efficiently rejecting candidates while using up the least amount of your existing people's time. Which means that the most expensive filters should be done last. And the most expensive one is social because judging it takes time from EVERYONE.
And that seems fair.
Why is it only the employee that is expected to take up time? At the end of the day there is likely to be more than one applicant, so your spending 3 hours of your time for 50% chance at best.
Just in case you look at your old thread.
The fundamental reason why this is OK is that the company is hoping to pay hundreds of thousands of dollars for the candidate's time. The person who pays the piper, gets to choose how it happens.
But that said, it is still fair. The effort required to hire a new person into a competent company is significantly greater than the effort it takes a competent person to get a new job. As a candidate it doesn't look like this because you see that you personally gave the company more attention than vice versa. However you don't see all of the other people who the company also paid attention to and ultimately rejected.
If you're a competent developer it probably still doesn't look like this because the company usually develops procedures that concentrate the required effort in the hiring manager and/or HR. Therefore you have little idea how much effort is actually spent looking for candidates.
But spend time as a hiring manager and it will be obvious.
The fundamental reason why this is OK is that the company is hoping to pay hundreds of thousands of dollars for the candidate's time. The person who pays the piper, gets to choose how it happens.
But that said, it is still fair. The effort required to hire a new person into a competent company is significantly greater than the effort it takes a competent person to get a new job. As a candidate it doesn't look like this because you see that you personally gave the company more attention than vice versa. However you don't see all of the other people who the company also paid attention to and ultimately rejected.
If you're a competent developer it probably still doesn't look like this because the company usually develops procedures that concentrate the required effort in the hiring manager and/or HR. Therefore you have little idea how much effort is actually spent looking for candidates.
But spend time as a hiring manager and it will be obvious.
Oh as a number of people have indicated
its bad for the company because the candidate can work with someone else and produce a glowing submission. I have helped a number of people do these some that have gotten the job.
its bad for the interviewee. They may spend 10-15 hours on this and get rejected for no reason at all. Do you think the person reviewing the submission is putting multiple hours into it. Doubtful.
If the intent is truly(and I mean truly) help individuals that struggle with traditional interviewing techniques then kudos. Demonstrate this by allowing candidates to interview in a way that is comfortable for them(including not taking your take home test) If its the company saying "my time is more valuable than yours. Do this assignment then we'll talk" then no thanks.
its bad for the company because the candidate can work with someone else and produce a glowing submission. I have helped a number of people do these some that have gotten the job.
its bad for the interviewee. They may spend 10-15 hours on this and get rejected for no reason at all. Do you think the person reviewing the submission is putting multiple hours into it. Doubtful.
If the intent is truly(and I mean truly) help individuals that struggle with traditional interviewing techniques then kudos. Demonstrate this by allowing candidates to interview in a way that is comfortable for them(including not taking your take home test) If its the company saying "my time is more valuable than yours. Do this assignment then we'll talk" then no thanks.
If you were able to explain your 'cheated' solution to someone and they could reliably cheat the interviewer, I'd say you .. taught them. GJ!
The interviewee (the vowel count makes me dizzy) can choose what he/she prefers. So that argument is weird. Maybe (if that's your point) it might be beneficial to go for the face to face interview, for direct feedback? But .. some people just _cannot_ do that. You're not talking "Ah, they didn't respond. If you'd just had visited them on-site..." here. It's "opt for the home project or don't apply". Which is empowering the candidate.
The interviewee (the vowel count makes me dizzy) can choose what he/she prefers. So that argument is weird. Maybe (if that's your point) it might be beneficial to go for the face to face interview, for direct feedback? But .. some people just _cannot_ do that. You're not talking "Ah, they didn't respond. If you'd just had visited them on-site..." here. It's "opt for the home project or don't apply". Which is empowering the candidate.
I think you are missing the point. If you are ok with a "Cheated" response then you are really hiring a candidate based on their network of experts. An equally or more qualified candidate could submit a less stellar answer only because their dad isn't an emeritus chair at a University.
I absolutely agree with your linked comment - I love these small tasks; even when I get rejected afterwards I still enjoy having done them.
Or, if they don't actually expect you to spend more than 3 hours on the task, they will compare you to people who have spent 9 hours on it.
Had that recently: they said "spend no more than two hours on it" so I finished in about an hour and 45 that evening and flipped it over.
Then they started asking why my 20 or so unit tests only covered the basics when other candidates had full unit tests in the two hour time frame. I told them the other candidates were simply lying :-)
Had that recently: they said "spend no more than two hours on it" so I finished in about an hour and 45 that evening and flipped it over.
Then they started asking why my 20 or so unit tests only covered the basics when other candidates had full unit tests in the two hour time frame. I told them the other candidates were simply lying :-)
If a company emphasizes quantity over quality during the interview process, run away.
Very true - it is part of the bane of take home assignments. Even for someone who can code extremely fast, a lot of the exams aren't realistic with time, and then they are compared with those done by people who put in multiples of the asked hours.
I rather put in those extra hours contracting (worst scenario), doing open source work (lots to be done that advances the productivity of tens of thousands of developers, if not more), into my hobbies, or into my friends.
I rather put in those extra hours contracting (worst scenario), doing open source work (lots to be done that advances the productivity of tens of thousands of developers, if not more), into my hobbies, or into my friends.
I would rather spend 8 hours at home giving the interviewer a realistic impression of my abilities than 1 hour mumbling over a whiteboard and getting trashcanned because the interviewer mistakenly thinks that whiteboard skills prove anything at all.
As for receiving outside help, if you're judging applicants solely on their homework results, you are Doing It Wrong. The best interview I've ever had gave me a take-home programming assignment by email, then when I came in the lead programmer asked me to explain the program line-by-line and justify various decisions I made. I got the job.
As for receiving outside help, if you're judging applicants solely on their homework results, you are Doing It Wrong. The best interview I've ever had gave me a take-home programming assignment by email, then when I came in the lead programmer asked me to explain the program line-by-line and justify various decisions I made. I got the job.
Speaking as an employer: we address this by keeping our coding exercise short, and specifying a time limit. The limit ("60-90 minutes") is not rigidly enforced, but candidates know we're able to see how long they spent, and in practice almost everyone spends more than 60 and less than 90 minutes.
It's tricky. An in-person test would put the candidate in an unfamiliar environment, and makes many people nervous. A take-home test without a time limit opens itself up to the "I'd better spend lots of extra time so I can look impressive" problem. A take-home test with a hard time limit can also make people nervous. This is the best compromise we've been able to come up with (suggestions for improvement welcomed!).
As suggested on this thread, we also don't ask candidates to tackle the exercise until they've had a chance to talk to us on the phone.
It's tricky. An in-person test would put the candidate in an unfamiliar environment, and makes many people nervous. A take-home test without a time limit opens itself up to the "I'd better spend lots of extra time so I can look impressive" problem. A take-home test with a hard time limit can also make people nervous. This is the best compromise we've been able to come up with (suggestions for improvement welcomed!).
As suggested on this thread, we also don't ask candidates to tackle the exercise until they've had a chance to talk to us on the phone.
This is very much true. We do take-home interview questions where they bring the output to the interview and we talk through their solution in person. We'll ask how long they took as a way to normalize expectations and also adjust the interview question to take more/less time in future instances.
This does two things. In the long run, the time required converges towards where we want it to be (ASSUMING honest answers), and by bringing you in to talk through things, we can easily tell if these thoughts were your own by challenging you on specific parts of the prompt.
This is less than ideal since it costs the interviewee time, and time isn't cheap, but we've found that advance prep removes an even bigger wildcard in interviews: how you respond to interview stress.
This does two things. In the long run, the time required converges towards where we want it to be (ASSUMING honest answers), and by bringing you in to talk through things, we can easily tell if these thoughts were your own by challenging you on specific parts of the prompt.
This is less than ideal since it costs the interviewee time, and time isn't cheap, but we've found that advance prep removes an even bigger wildcard in interviews: how you respond to interview stress.
Tip: If you want honest answers about how long it took, ask them after they've been hired and working there for a few weeks. Otherwise they will absolutely lie.
"That? Oh, it was no big, probably 20-30 minutes."
"That? Oh, it was no big, probably 20-30 minutes."
Honestly I kind of enjoy take home tests as long as they align with what I'm learning and looking to learn. I've had to do a couple take home interviews as well at this point as part of the initial phone screen, although they revolve more around config management tools/Ruby DSL and buildout work so far. So far the tests I've taken have been relevant to what I'm hoping to move into ("DevOps"/automation) so they've been a pleasure to complete. Not only that, but also a base to build on for playing with additional tools in the chain.
The biggest problem with these though is definitely the time crunch. The first one I hadn't realized how long it would take so rushed through it at the end and made mistakes. Second one gave myself a full week rather than 4 days and that's proceeding to an in-person interview, so you need to be taking as much time as they'll allow in order for it to go smoothly.
Also, another project means more documentation added to the repo so that's pretty nice too. If the project didn't align to my interests, which thankfully happen to be stupidly in-demand right now if you have "senior" experience with them, and the position, I'd nope out of the project right away.
The biggest problem with these though is definitely the time crunch. The first one I hadn't realized how long it would take so rushed through it at the end and made mistakes. Second one gave myself a full week rather than 4 days and that's proceeding to an in-person interview, so you need to be taking as much time as they'll allow in order for it to go smoothly.
Also, another project means more documentation added to the repo so that's pretty nice too. If the project didn't align to my interests, which thankfully happen to be stupidly in-demand right now if you have "senior" experience with them, and the position, I'd nope out of the project right away.
> I've seen applicants receive friends'/roommates'/spouse's help on take home tests. Not a good indicator at all even with a glowing submission.
That's why they use the homework as the basis for the interview. In my opinion, of all things they could test you on during the interview, your own work is potentially one of the more pleasant subjects.
That's why they use the homework as the basis for the interview. In my opinion, of all things they could test you on during the interview, your own work is potentially one of the more pleasant subjects.
I've had a few take home projects that weren't so bad.
But the last one I did...
A junior just basically shat all over the project and claimed a lot of things.
[The submission was to write a few sample sort functions for a library... the recruiter asked for an app, the paper asked for a library.. I did both... what did I get shat upon for? The user interface/cli that wasn't required.. another thing.. Why did I have 66 commits?]
But the last one I did...
A junior just basically shat all over the project and claimed a lot of things.
[The submission was to write a few sample sort functions for a library... the recruiter asked for an app, the paper asked for a library.. I did both... what did I get shat upon for? The user interface/cli that wasn't required.. another thing.. Why did I have 66 commits?]
yuck! why did you have 66 commits?
What's wrong with having 66 commits?
This is a completely generic response which is not at all tailored to the above poster's specific situation (since I'm not familiar with the details), but the general idea would be something like this:
If you asked someone to do FizzBuzz at home and they sent you a git repo with 60+ commits, you wouldn't be a little puzzled about that? That would seem to suggest they really struggled with it.
If you asked someone to do FizzBuzz at home and they sent you a git repo with 60+ commits, you wouldn't be a little puzzled about that? That would seem to suggest they really struggled with it.
That's not really a fair comparison though. It would be more like a typical "whiteboard coding" problem needing a couple commits.
I'm not really sure what you mean. We don't know what the task was that he had to solve. I was providing an example of a situation in which 66 commits could be seen as a bad thing.
For a typical whiteboard coding problem, I also wouldn't want to see 60+ commits.
For a typical whiteboard coding problem, I also wouldn't want to see 60+ commits.
I remember when I first got started with git I committed at an absurd frequency. I don't think it's that unreasonable for someone straight out of college to be relatively new to git, and thus do the same thing.
I remember later on I wrote a script that listened to git hooks and rebuilt my project on a remote server. I was still testing manually at that time, as we all do in the beginning, which resulted in a large number of commits so that I could view the results on the server.
I think it's ok to ask "why do you have so many commits?" but not "why do you have so many commits?!?!?!". It's also not ok to ASSUME that a large number of commits is a bad thing automatically, unless you have reasons far better then any submitted in this thread.
I remember later on I wrote a script that listened to git hooks and rebuilt my project on a remote server. I was still testing manually at that time, as we all do in the beginning, which resulted in a large number of commits so that I could view the results on the server.
I think it's ok to ask "why do you have so many commits?" but not "why do you have so many commits?!?!?!". It's also not ok to ASSUME that a large number of commits is a bad thing automatically, unless you have reasons far better then any submitted in this thread.
Well I can't say too much without exposing the problem.
But there were about 16 files in total. All of the comparators that were written had multiple tests. (I shot for nearly 100% coverage..although I wasn't going to force stdin emulation, and mock out system.exits) All of the commits that were performed were done in small amounts. (Also, a few were just for transferring work space to other computers) The task in hand would have represented approximately 4 tickets at the bare minimum.
Besides, the number of commits: That problem is resolved by merging a branch back down.
The feedback was written in a way "How do you consider that reasonable in a professional environment?"
(Another thing... I wasn't required to submit via a git repo I was required to submit via a zip file, but I did because I hoped it'd give me a leg up.... Turns out: You're better off not trying)
But there were about 16 files in total. All of the comparators that were written had multiple tests. (I shot for nearly 100% coverage..although I wasn't going to force stdin emulation, and mock out system.exits) All of the commits that were performed were done in small amounts. (Also, a few were just for transferring work space to other computers) The task in hand would have represented approximately 4 tickets at the bare minimum.
Besides, the number of commits: That problem is resolved by merging a branch back down.
The feedback was written in a way "How do you consider that reasonable in a professional environment?"
(Another thing... I wasn't required to submit via a git repo I was required to submit via a zip file, but I did because I hoped it'd give me a leg up.... Turns out: You're better off not trying)
66 commits sounds perfectly fine for complete unit test coverage, 16 files, etc. Sounds like this was not the sort of person anybody would want to work with if he phrased the question in that condescending way. He'd probably end up being an insane micromanager.
[deleted]
For a whiteboard coding problem, you're really not going to see more than a function. So I completely agree with you.
I meant saying Fizzbuzz with 66 commits is not a fair comparison to a library having 66 commits.
>Schedule 2x or 3x the estimate.
This might even be an underestimate. In my experience, there is a lot of research time that goes into the problem before you really dig in and start coding. I had one take-home project that required a few days just to get my system configured to begin testing code (collecting the dataset [10's of GB], installing libraries, and configuring the system).
>I've seen applicants receive friends'/roommates'/spouse's help on take home tests.
I enjoy overtly mathematical problems, so I've talked through a number of take-home interview questions with friends during the research phase, and even provided implementations for comparison and review after they've returned them. (Most recently on a take-home challenge to generate digits of pi.)
>But the company just took 10 minutes to arbitrarily reject your application.
This is actually my second biggest gripe as an applicant. I spend a couple of hours building and submitting the most compelling application I can for a job. The worst so far was an automated email response that my application had been forwarded for review, and before I finished reading the automated response I got a rejection email from the hiring manager. The emails are literally two minutes apart in my inbox. It is incredibly frustrating to put so much time into applications when they are clearly being summarily rejected.
(For anyone curious, my biggest job search gripe is not receiving any kind of firm decision...ever. I can appreciate that there "is not a good fit at this time", but I'm not going to be sitting here in six months pining for that job I applied for with your company. If I want to show interest, I'll apply again in a year or so. It's actually mildly frustrating when I get a callback three months later.)
This might even be an underestimate. In my experience, there is a lot of research time that goes into the problem before you really dig in and start coding. I had one take-home project that required a few days just to get my system configured to begin testing code (collecting the dataset [10's of GB], installing libraries, and configuring the system).
>I've seen applicants receive friends'/roommates'/spouse's help on take home tests.
I enjoy overtly mathematical problems, so I've talked through a number of take-home interview questions with friends during the research phase, and even provided implementations for comparison and review after they've returned them. (Most recently on a take-home challenge to generate digits of pi.)
>But the company just took 10 minutes to arbitrarily reject your application.
This is actually my second biggest gripe as an applicant. I spend a couple of hours building and submitting the most compelling application I can for a job. The worst so far was an automated email response that my application had been forwarded for review, and before I finished reading the automated response I got a rejection email from the hiring manager. The emails are literally two minutes apart in my inbox. It is incredibly frustrating to put so much time into applications when they are clearly being summarily rejected.
(For anyone curious, my biggest job search gripe is not receiving any kind of firm decision...ever. I can appreciate that there "is not a good fit at this time", but I'm not going to be sitting here in six months pining for that job I applied for with your company. If I want to show interest, I'll apply again in a year or so. It's actually mildly frustrating when I get a callback three months later.)
I've gotten some pretty wacky projects. Basically, implement a fairly-complex web app. Something that I'd assume (even after doing similar tasks) would probably take me up to 40 hours of work, just because the scope is so crazy. They want user login, multiple features, etc... basically an MVP for a small product.
It's hard. If you submit part of the project, which you usually have to do unless you're unemployed, they never go for it. If you call out of work a day or two to do the project... and you don't get the job anyway...
It's hard. If you submit part of the project, which you usually have to do unless you're unemployed, they never go for it. If you call out of work a day or two to do the project... and you don't get the job anyway...
I recently interviewed for 4 jobs, one of them was one of these type of "you must implement this in 1 hour". It was painful. I didn't finish in time. Apparently the idea was you much choose language X, it will e pretty easy with language X. I chose Y, and they said why did you choose Y, that was stupid. Result - I chose one of my other 3 offers, took the one that made me a staff programmer. And I live in Seattle, ha ha bay area.
[deleted]
My wife is not a technologist. In fact, she's a biologist & nurse who's worked in pharmacoviligance for a series of pharmas & CROs past ~15 years. She was laid off this spring after her company was purchased and went through the whole interview thing. One company had a writing assignment as part of their process. It was after both the phone screening and the in person interviews, and they only gave it to candidates they really liked/wanted, AND it was literally a mini version of the kind of data analysis and reports[1] she'd be doing in the job.
But ... it was something that took about 40 hours to complete, and ruined about a week and a half of family time. Thankfully she got the job, but if she hadn't it wouldn't have been pretty.
[1] http://onbiostatistics.blogspot.com/2013/07/periodic-safety-...
But ... it was something that took about 40 hours to complete, and ruined about a week and a half of family time. Thankfully she got the job, but if she hadn't it wouldn't have been pretty.
[1] http://onbiostatistics.blogspot.com/2013/07/periodic-safety-...
As an outsider, that sets off red flags for me. Sounds suspiciously like unpaid work rather than a take home assignment.
I agree, the solution is terrible, but the problem is real. Whiteboarding is an awful way to measure someone's skills.
I've had an opposite experience, but with just one company: CloudMine. They gave me a personal and technical interview, then a project to add a feature to their existing codebase. They paid me $300 for my work. I was way over time estimates and I didn't get the job, but I will always respect them for that.
The company I work for has a "coding assignment," and it works really well for us. I'm not sure we give a time estimate; I think we just ask for it back in a week or so. The devs reviewing the code don't even know how long it took for the applicant to return, and it isn't a criteria we use to judge them. In any case, I can't imagine it would take anybody more than a few hours.
IMO, it's a less insulting, slightly more realistic version of FizzBuzz.
As far as getting help goes, I'm not sure it matters too much. There's still an on site interview, and we ask a few questions about the assignment, along with some other coding questions, and regular interview stuff. I guess if their friend/roommate/spouse is going to help them code all the time, then it's like we hired two people for the price of one ;-)
IMO, it's a less insulting, slightly more realistic version of FizzBuzz.
As far as getting help goes, I'm not sure it matters too much. There's still an on site interview, and we ask a few questions about the assignment, along with some other coding questions, and regular interview stuff. I guess if their friend/roommate/spouse is going to help them code all the time, then it's like we hired two people for the price of one ;-)
We do take-home tests at deezer.
We try hard to provide good feedback on each submission (certainly not just a 10 minutes look, even for a truly awful entry).
I think that we should try harder to provide an exercise with a maximum time spent. We also try hard to overlook things that can be attributed to a lack of time but the openness of the exercises we use mean that somebody willing to could spend dozens of hours on it if he/she wanted a truly perfect solution (with unit tests everywhere, handling all API levels & devices in the wild, ...)
I personally thinks that our exercises are more objective than whiteboard tests & better reflect real life work. Actually, we take our inspiration from real problems we had to solve.
We try hard to provide good feedback on each submission (certainly not just a 10 minutes look, even for a truly awful entry).
I think that we should try harder to provide an exercise with a maximum time spent. We also try hard to overlook things that can be attributed to a lack of time but the openness of the exercises we use mean that somebody willing to could spend dozens of hours on it if he/she wanted a truly perfect solution (with unit tests everywhere, handling all API levels & devices in the wild, ...)
I personally thinks that our exercises are more objective than whiteboard tests & better reflect real life work. Actually, we take our inspiration from real problems we had to solve.
If you only supposed to take three hours why not set it up like a take home test? Set up a site where you can download it and then you have three hours to upload the answer?
Another problem is that its not a zero sum game candidates are applying to multiple potential employers.
Forcing you to spend most of a day on an application means that good candidates will just not wast time applying for these sorts of jobs.
An in the uk to get befits you have to provide evidence of applying for x no of jobs per week so wasting 2 days on a single application is out
Forcing you to spend most of a day on an application means that good candidates will just not wast time applying for these sorts of jobs.
An in the uk to get befits you have to provide evidence of applying for x no of jobs per week so wasting 2 days on a single application is out
> Take home tests are the worst. Company says take home test will take 3 hours to complete. They never do. Schedule 2x or 3x the estimate. Especially if you want to impress the reviewer.
So only do timed tests?
So only do timed tests?
My co-worker once found that someone had posted the take home problem on one of those freelance job sites. Needless to say we didn't move forward with the candidate.
If they can get the take home problem solved for a reasonable cost and in short time on the freelance job site, hire them as a manager.
I'd even say: if they can get the problem solved for less than their salary and in reasonable time, hire them anyway - you're hiring them to solve programming problems, not to type them.
Also a lot of the take home interviews are time limited. So you would only have a certain amount of time to implement a solution.
No, in the worst case, you submit it and the company doesn't even take a look at it. That's not only the worst case, it seems to be the most common as well.
Tech recruiter here:
I have seen so many companies that lose out on good developers because they ask them to do a take home test.
Obviously you want to vet the person technically but you should be able to do that through talking with them. Most developers put off take home tests even if they are excited about the company and by the time they do it they already have finals/offers from other companies.
Obviously you want to vet the person technically but you should be able to do that through talking with them. Most developers put off take home tests even if they are excited about the company and by the time they do it they already have finals/offers from other companies.
Why don't interviewers just have a conversation or is that so totally lost these days? I find it a whole lot easier to just ask the right technical questions in the first 2 mins. to decide on hiring.
I keep seeing more and more dis-associative non-communication being presented where the interviewer attempts to do less and less. Its lazy and a crap way to treat people.
I keep seeing more and more dis-associative non-communication being presented where the interviewer attempts to do less and less. Its lazy and a crap way to treat people.
I usually refuse them unless I have spoken with the technical people first. You get no idea about the job from HR staff.
I dont really understand the way interviewing is set up in general. Interviewing in my mind is a two-way street. It’s not just the candidate that’s being interviewed for the job, it’s also the employer being interviewed to determine suitability.
In light of this, I think most interview processes don’t make any sense. I think that a screening phone/Skype call makes sense before either side commits any time/resources etc. In this process, I think there should always be room for the candidate to ask questions of the employer. Any situation that is unbalanced, in which the employer is playing the role of the one with the final decision is a completely skewed setup.
I interviewed a while back with a YC company and although I didn’t get the job, I was highly appreciative of the setup. Firstly, there was a Skype/phone call, which felt really like an open conversation; one in which I was free to ask questions and understand suitability too.
Then I was invited to an in-house hack. Essentially, I spent the day (a Saturday) hacking with the founders on a project. The project was a lot of fun and something that I learned a ton from. We actually ended up working till around 4am the next day. What I really appreciated is that although it was “an exam” for me, the investment of time on my part was weighed equally by their time investment. Also, I was happy to know that they had learned things during the hack too. I was at ease during the whole day precisely because they treated me as though I was their colleague already.
Coming out of that experience, I’ve wondered for a while why employers don’t take the stance that to hire excellence you have to commit time and resources. Most interview processes that I’ve heard of seem to be a cop out on the part of the recuiter/interviewer, in the sense that it seems that the priority is to hire the best people but spend the least time doing it. That to me is the ingredient for a broken process.
So, in a nutshell, I think talk of take-home interviews misses the point. It might not be scalable, but my intuition tells me that it would be much more effective to have the interviewee and company folks hack on something together. That way, the interviewee gets the opportunity to essentially interview the company too. I’d definitely preceed this with a few conversations on the phone/in person, to ensure that the time investment of hacking together is geared towards the best interviewee/company relationships.
Perhaps this is missing the point too, and perhaps it’s not workable in the real world. My anecdoctal evidence is simply that have spent an entire Saturday working on a project as part of an interview process and ultimately not getting the job, I don’t feel I wasted any time, nor do I feel bitter about it.
In light of this, I think most interview processes don’t make any sense. I think that a screening phone/Skype call makes sense before either side commits any time/resources etc. In this process, I think there should always be room for the candidate to ask questions of the employer. Any situation that is unbalanced, in which the employer is playing the role of the one with the final decision is a completely skewed setup.
I interviewed a while back with a YC company and although I didn’t get the job, I was highly appreciative of the setup. Firstly, there was a Skype/phone call, which felt really like an open conversation; one in which I was free to ask questions and understand suitability too.
Then I was invited to an in-house hack. Essentially, I spent the day (a Saturday) hacking with the founders on a project. The project was a lot of fun and something that I learned a ton from. We actually ended up working till around 4am the next day. What I really appreciated is that although it was “an exam” for me, the investment of time on my part was weighed equally by their time investment. Also, I was happy to know that they had learned things during the hack too. I was at ease during the whole day precisely because they treated me as though I was their colleague already.
Coming out of that experience, I’ve wondered for a while why employers don’t take the stance that to hire excellence you have to commit time and resources. Most interview processes that I’ve heard of seem to be a cop out on the part of the recuiter/interviewer, in the sense that it seems that the priority is to hire the best people but spend the least time doing it. That to me is the ingredient for a broken process.
So, in a nutshell, I think talk of take-home interviews misses the point. It might not be scalable, but my intuition tells me that it would be much more effective to have the interviewee and company folks hack on something together. That way, the interviewee gets the opportunity to essentially interview the company too. I’d definitely preceed this with a few conversations on the phone/in person, to ensure that the time investment of hacking together is geared towards the best interviewee/company relationships.
Perhaps this is missing the point too, and perhaps it’s not workable in the real world. My anecdoctal evidence is simply that have spent an entire Saturday working on a project as part of an interview process and ultimately not getting the job, I don’t feel I wasted any time, nor do I feel bitter about it.
This is great. I especially love that candidates are given a choice, so if they prefer the traditional technical interview they can choose that.
Normalizing performance between the two interview types will be challenging, but I think the benefits to be reaped far outweigh the difficulty of the challenges.
Normalizing performance between the two interview types will be challenging, but I think the benefits to be reaped far outweigh the difficulty of the challenges.
Agreed. My reaction was "finally!". I really do not approach software engineering as Performance Art -- and I never write code on white boards when I work. I've been fortunate to have been asked only once to do such a performance in an interview, and my reaction was unfortunately (and unexpectedly) like the candidate they described. In that case I didn't write anything on the board, but instead explained verbally how I would approach the problem. They actually gave me a "take home" problem to solve. I was able to go to my office and produce a solution like I normally do, and then sent them the sources and test results.
Very glad to hear that folks are breaking free of this long-lived trend. I also interview candidates on behalf of my clients. Personally, I find the most effective technique is to choose items from their CV, and have describe in detail (with a white board) how they did it, what challenges they faced, etc. Then, pose to them a hypothetical "but what if you were constrained by X or Y, how would you adapt your solution", etc. It is very easy to spot a charlatan or liar in these questions. If they really did what they claimed on the CV, this type of session gives them great latitude to demonstrate and expand on what their capabilities are.
Very glad to hear that folks are breaking free of this long-lived trend. I also interview candidates on behalf of my clients. Personally, I find the most effective technique is to choose items from their CV, and have describe in detail (with a white board) how they did it, what challenges they faced, etc. Then, pose to them a hypothetical "but what if you were constrained by X or Y, how would you adapt your solution", etc. It is very easy to spot a charlatan or liar in these questions. If they really did what they claimed on the CV, this type of session gives them great latitude to demonstrate and expand on what their capabilities are.
You've got to be a little careful picking out items from their CV, though, especially if you go back far enough. I barely remember anything I did six months ago, and anything one or two years ago I couldn't explain to you in anything more than a really broad overview, despite being neck deep in the code. I've since moved past that, tackled other projects, stuffed my head full of knowledge of other systems (and hobbies), and all that deep technical information on those projects is long, long gone.
If you put it on your resume, its fair game for the interviewer to ask about it.
Not disagreeing with you, but just saying there can be legitimate reasons why they don't have deep knowledge on past projects besides "they're a total charlatan, a liar!" It's probably cost me jobs before, and I try to reflect on and refresh my memory on past jobs as best as I can before interviews usually.
I've even gone over notes I took at a couple of my past jobs, and I don't even remember doing a lot of those things even with it written down in my own handwriting.
I've even gone over notes I took at a couple of my past jobs, and I don't even remember doing a lot of those things even with it written down in my own handwriting.
That looks great. Are there enough problem seeds to help combat sharing of answers?
Now we just need you at non-YC startups, too.
Now we just need you at non-YC startups, too.
Triplebyte founder here. I can't be entirely sure until we give this a try, but the hope is we'll be able to tell if the candidate wrote the code by talking to them for 45-minutes about what they've done.
Seems good, I like the choice.
In discussions on this topic I see a lot of: "Programming on the spot is hard, let people program at home"! But then other people say "Why should I program for free at home, my resume clearly shows I am already a skilled programmer. All this will do is cater to young people without families, or those fresh out of school".
I am not looking to hire devs right now, but I am thinking about the same idea with a tiny tweak when we move to that stage. If the verbal interview goes well, offer 2 choices. Make it clear neither choice is seen as the better choice, and both will be judged equally.
* Sit on a computer with me for an hour, and show me how to code a fairly simple program.
or
* Get a small actual work assignment to take home and code. Would expect it to take maybe 5 hours to code. Offer $500 as a 1099 contractor to complete the assignment within the next 2 weeks or so.
This gives both groups a chance. The too busy to do a big programming assignments can code in front of me for an hour. I should be able to judge their chops pretty fast. For the people more nervous to code in front of me, they can get paid a nominal fee to code some small piece of code.
The only group I exclude is the group that doesn't feel the need to show code to land a job, but not too interested in that group. I have seen too many good talkers and bad coders to want to risk this group.
In discussions on this topic I see a lot of: "Programming on the spot is hard, let people program at home"! But then other people say "Why should I program for free at home, my resume clearly shows I am already a skilled programmer. All this will do is cater to young people without families, or those fresh out of school".
I am not looking to hire devs right now, but I am thinking about the same idea with a tiny tweak when we move to that stage. If the verbal interview goes well, offer 2 choices. Make it clear neither choice is seen as the better choice, and both will be judged equally.
* Sit on a computer with me for an hour, and show me how to code a fairly simple program.
or
* Get a small actual work assignment to take home and code. Would expect it to take maybe 5 hours to code. Offer $500 as a 1099 contractor to complete the assignment within the next 2 weeks or so.
This gives both groups a chance. The too busy to do a big programming assignments can code in front of me for an hour. I should be able to judge their chops pretty fast. For the people more nervous to code in front of me, they can get paid a nominal fee to code some small piece of code.
The only group I exclude is the group that doesn't feel the need to show code to land a job, but not too interested in that group. I have seen too many good talkers and bad coders to want to risk this group.
> Would expect it to take maybe 5 hours to code.
So, if you are interviewing at 20 companies, you need to expend 100+ hours or almost 3 full work weeks?
It should take 1 hour or less. Period. Probably less that 1/2 hour. I doubt I get much more information asking you for a 5 hour assignment than a 1/2 hour assignment.
If I'm really that interested in your programming on the fly, I should do it at the interview where I'm paying for your lodging, food, etc. I should tell you what you are going to be doing, and to bring your laptop set up to do that.
So, if you are interviewing at 20 companies, you need to expend 100+ hours or almost 3 full work weeks?
It should take 1 hour or less. Period. Probably less that 1/2 hour. I doubt I get much more information asking you for a 5 hour assignment than a 1/2 hour assignment.
If I'm really that interested in your programming on the fly, I should do it at the interview where I'm paying for your lodging, food, etc. I should tell you what you are going to be doing, and to bring your laptop set up to do that.
If you're interviewing with 20 companies, you're probably already spending 4+ hours with each of them in person.
That's a lot of companies to entertain! When I was interviewing, I could barely maintain the composure to interview with 3 companies! (Only one of which had a take-home assignment.)
That's a lot of companies to entertain! When I was interviewing, I could barely maintain the composure to interview with 3 companies! (Only one of which had a take-home assignment.)
> So, if you are interviewing at 20 companies, you need to expend 100+ hours or almost 3 full work weeks?
I have never interviewed at 20 companies when I needed a new position. If you do, cool - but not sure I want to simplify my hiring process to help someone mass interviewing.
If we have a conversation and we both like each other, let's see if we can come to terms we are both happy with. If we are playing other companies off each other for 1k raises, it seems not really worth the hassle.
I have never interviewed at 20 companies when I needed a new position. If you do, cool - but not sure I want to simplify my hiring process to help someone mass interviewing.
If we have a conversation and we both like each other, let's see if we can come to terms we are both happy with. If we are playing other companies off each other for 1k raises, it seems not really worth the hassle.
Its good in theory, but it leaves a lot of ambiguity for the interviewee to parse through. For instance, if given a two week window would it look better for me to do it tonight? Would it show ambition? Or would it seem like I'm desperate, and lead to a lower amount of compensation being offered to me? Should I spend far longer then five hours on the assignment, and turn in superior work while making it seem like I only spent 5 hours? What is my competition doing?
As someone who's recently been through the ringer, including one six hour take-home, all I want is a clear demonstration of respect and rationality.
Bring me into your professional office. Let's talk like professionals. Allow me to demonstrate my professional skills. Call me back with a professional yes or a professional no, all within a professional time frame. That's it.
Remote or otherwise less traditional assignments / roles / jobs will deserve and benefit from their own process. But how we strayed from the straight forward formula is beyond me, my guess it was an initiative started by a handful of companies who had trouble hiring.
I don't think hiring devs is the systematic issue everyone makes it out to be. Assessing talent is always hard, be it an artist's, an athlete's or a programmers. I think rather then assume the cost of the investment in hiring, companies chose to blame the system and that's where this absurd roller coaster started.
As someone who's recently been through the ringer, including one six hour take-home, all I want is a clear demonstration of respect and rationality.
Bring me into your professional office. Let's talk like professionals. Allow me to demonstrate my professional skills. Call me back with a professional yes or a professional no, all within a professional time frame. That's it.
Remote or otherwise less traditional assignments / roles / jobs will deserve and benefit from their own process. But how we strayed from the straight forward formula is beyond me, my guess it was an initiative started by a handful of companies who had trouble hiring.
I don't think hiring devs is the systematic issue everyone makes it out to be. Assessing talent is always hard, be it an artist's, an athlete's or a programmers. I think rather then assume the cost of the investment in hiring, companies chose to blame the system and that's where this absurd roller coaster started.
> Bring me into your professional office. Let's talk like professionals. Allow me to demonstrate my professional skills. Call me back with a professional yes or a professional no, all within a professional time frame. That's it.
What if your professional skills are such that it's not possible to demonstrate them in an hour or two at someone else's office--either because of the nature of the skills, or because you're one of the many excellent programmers whose work suffers when there's someone actively staring at you, like the guy in the article?
Let's be honest. Programming as a discipline attracts a disproportionate number of people for whom social skills do not come easily. Granted, you don't want to hire a grumbling misanthrope who refuses to take direction, but you also don't want to turn away a perfectly good team player who lacks the largely irrelevant skill of gladhanding under pressure.
I think a lot of the comments in discussions like this come from programmers who do have solid social skills, and there's certainly nothing wrong with that, but an interview process that gives you personally a fair evaluation is not necessarily the most reliable one for programmers in general.
What if your professional skills are such that it's not possible to demonstrate them in an hour or two at someone else's office--either because of the nature of the skills, or because you're one of the many excellent programmers whose work suffers when there's someone actively staring at you, like the guy in the article?
Let's be honest. Programming as a discipline attracts a disproportionate number of people for whom social skills do not come easily. Granted, you don't want to hire a grumbling misanthrope who refuses to take direction, but you also don't want to turn away a perfectly good team player who lacks the largely irrelevant skill of gladhanding under pressure.
I think a lot of the comments in discussions like this come from programmers who do have solid social skills, and there's certainly nothing wrong with that, but an interview process that gives you personally a fair evaluation is not necessarily the most reliable one for programmers in general.
Good points. If you really care about being impartial, maybe have the 2 week period and a blind submission method in which the interviewer does not see when the assignment was completed.
But my rent is due the Friday after next and I need to know if I should send out another wave of resumes.
It isn't but it strikes me we are looking very hard to find a new way to do things, when the old way was pretty damn good.
Sit me down and talk about technology for ~thirty minutes. If I don't have the social skills to successfully do this (minority issue) I likely would not be able to communicate well with a team and thus should not be a candidate anyways. You will then know immediately whether you want to hire me (or progress me to another round) or not.
I then get a call two days later and can progress with my life.
People pretend like all this hiring strategy is for the good of the candidate. It's not. As with everything else its for the good of the company and its investors.
Here's a thought:
Invest in a competent hiring manager who can see through applicant bull shit and identify talent within a reasonable range. Includes basic negotiation skills. And assume the rightful risk that is employing another human being.
It isn't but it strikes me we are looking very hard to find a new way to do things, when the old way was pretty damn good.
Sit me down and talk about technology for ~thirty minutes. If I don't have the social skills to successfully do this (minority issue) I likely would not be able to communicate well with a team and thus should not be a candidate anyways. You will then know immediately whether you want to hire me (or progress me to another round) or not.
I then get a call two days later and can progress with my life.
People pretend like all this hiring strategy is for the good of the candidate. It's not. As with everything else its for the good of the company and its investors.
Here's a thought:
Invest in a competent hiring manager who can see through applicant bull shit and identify talent within a reasonable range. Includes basic negotiation skills. And assume the rightful risk that is employing another human being.
> Sit me down and talk about technology for ~thirty minutes.
This is an efficient way to hire a team of good bullshitters. I've interviewed people who did extremely well when we were "talking like professionals" but were unable to do even very simple coding problems.
My bar for coding is really not that high. I don't expect perfect syntax. I don't pick the language. I don't expect "the one answer". I expect people to write code that could work after syntax and small bugs are fixed, and most critically, I expect people to be able to discuss their code meaningfully.
Unfortunately, "ability to write basic code" and "ability to talk like professionals" are not tightly correlated. And I expect both of these things from dev candidates.
This is an efficient way to hire a team of good bullshitters. I've interviewed people who did extremely well when we were "talking like professionals" but were unable to do even very simple coding problems.
My bar for coding is really not that high. I don't expect perfect syntax. I don't pick the language. I don't expect "the one answer". I expect people to write code that could work after syntax and small bugs are fixed, and most critically, I expect people to be able to discuss their code meaningfully.
Unfortunately, "ability to write basic code" and "ability to talk like professionals" are not tightly correlated. And I expect both of these things from dev candidates.
I disagree. I have yet to meet anyone who is faking it and cannot be cracked in 5-10 minutes of carefully-directed prodding. In fact, I don't see how it is possible to do extremely well on the "talk like a professional" part and not be able to write basic programs, unless you and I have very different concepts of what "talk like a professional" means.
> I have yet to meet anyone who is faking it and cannot be cracked in 5-10 minutes of carefully-directed prodding.
Coding is a skill largely separate from other critical dev skills, e.g. design. Someone can legitimately have deep knowledge of, e.g., how to build a 3-tier service architecture without actually being able to code. This isn't "faking it". It's a different skill. You can study database partitioning strategies and learn about the latest and greatest frameworks for writing web pages and APIs without ever writing a single line of code. And sure, you can try to probe deeply enough to expose where their knowledge stops, but you're still not probing for coding skill. There's actually a decent chance that you'll end up probing for trivia which is not a very useful filter. "Oh, you don't don't know the auto-generated C++ class members? You clearly haven't actually worked in C++!" Yeah, right. Anything known broadly-enough known that it's reasonable to assume all coders would know it is broadly-enough known that non-coders could learn it, too.
Even if you could probe deeply enough in conversation to expose that they can't code, you could do the same with a few minutes of actual coding. When someone can't code, it becomes clear pretty quickly when you ask them to code.
Coding is a skill largely separate from other critical dev skills, e.g. design. Someone can legitimately have deep knowledge of, e.g., how to build a 3-tier service architecture without actually being able to code. This isn't "faking it". It's a different skill. You can study database partitioning strategies and learn about the latest and greatest frameworks for writing web pages and APIs without ever writing a single line of code. And sure, you can try to probe deeply enough to expose where their knowledge stops, but you're still not probing for coding skill. There's actually a decent chance that you'll end up probing for trivia which is not a very useful filter. "Oh, you don't don't know the auto-generated C++ class members? You clearly haven't actually worked in C++!" Yeah, right. Anything known broadly-enough known that it's reasonable to assume all coders would know it is broadly-enough known that non-coders could learn it, too.
Even if you could probe deeply enough in conversation to expose that they can't code, you could do the same with a few minutes of actual coding. When someone can't code, it becomes clear pretty quickly when you ask them to code.
> Coding is a skill largely separate from other critical dev skills, e.g. design.
I do not believe coding is largely separate from engineering. There are, of course, other skills that are necessary or desirable.
> Someone can legitimately have deep knowledge of, e.g., how to build a 3-tier service architecture without actually being able to code. This isn't "faking it". It's a different skill.
Absolutely. It is also a level of skill I would only expect or look for in a position that was not going to be doing daily coding, such as a systems engineer.
> You can ... learn about the latest and greatest frameworks for writing web pages and APIs without ever writing a single line of code.
Here I disagree. If you have never used a framework or API, you don't know it. I don't care that you can regurgitate the Javadocs; I care that you know things like its pitfalls, quirks, and runtime oddities.
> There's actually a decent chance that you'll end up probing for trivia which is not a very useful filter.
To be clear, I abhor trivia-based interviewing. What I am talking about is practical engineering tradeoffs between one course of action versus another, preferably supported by direct personal experience in the past.
> Even if you could probe deeply enough in conversation to expose that they can't code, you could do the same with a few minutes of actual coding. When someone can't code, it becomes clear pretty quickly when you ask them to code.
Exposing the level of competence a five-minute coding problem can is trivial to do in parallel with a deep engineering discussion. In fact, I think some sort of code should be part of that discussion. I just don't think it should be the "implement this" CLRS problem. It should be something two-way, more representative of what the job is like on a daily basis.
I do not believe coding is largely separate from engineering. There are, of course, other skills that are necessary or desirable.
> Someone can legitimately have deep knowledge of, e.g., how to build a 3-tier service architecture without actually being able to code. This isn't "faking it". It's a different skill.
Absolutely. It is also a level of skill I would only expect or look for in a position that was not going to be doing daily coding, such as a systems engineer.
> You can ... learn about the latest and greatest frameworks for writing web pages and APIs without ever writing a single line of code.
Here I disagree. If you have never used a framework or API, you don't know it. I don't care that you can regurgitate the Javadocs; I care that you know things like its pitfalls, quirks, and runtime oddities.
> There's actually a decent chance that you'll end up probing for trivia which is not a very useful filter.
To be clear, I abhor trivia-based interviewing. What I am talking about is practical engineering tradeoffs between one course of action versus another, preferably supported by direct personal experience in the past.
> Even if you could probe deeply enough in conversation to expose that they can't code, you could do the same with a few minutes of actual coding. When someone can't code, it becomes clear pretty quickly when you ask them to code.
Exposing the level of competence a five-minute coding problem can is trivial to do in parallel with a deep engineering discussion. In fact, I think some sort of code should be part of that discussion. I just don't think it should be the "implement this" CLRS problem. It should be something two-way, more representative of what the job is like on a daily basis.
> I do not believe coding is largely separate from engineering.
I could pick up a cookbook and study how to cook a souffle. I could learn enough about this academically that I could answer basically any questions you might ask. Having this academic knowledge is a good thing, but it doesn't mean I've ever even separated an egg. If you really want to know if I can make a souffle, your best bet is to hand me the stuff and ask me to do it.
> Here I disagree. If you have never used a framework or API, you don't know it. I don't care that you can regurgitate the Javadocs; I care that you know things like its pitfalls, quirks, and runtime oddities.
And here I assert that you're either probing on trivia or probing on things that can be learned without using the framework (actually, probably both). The pitfalls, quirks, and oddities are well documented in thousands of blogs.
> To be clear, I abhor trivia-based interviewing. What I am talking about is practical engineering tradeoffs between one course of action versus another, preferably supported by direct personal experience in the past.*
So this is an entirely different thing than pitfalls. Now you're talking about designing systems and dealing with tradeoffs. This isn't coding, and I don't believe you can always discover coding gaps by probing on this.
> Exposing the level of competence a five-minute coding problem can is trivial to do in parallel with a deep engineering discussion. In fact, I think some sort of code should be part of that discussion.
I guess I'm confused about what you're asking in an interview, then. Are your candidates coding or not?
> It should be something two-way, more representative of what the job is like on a daily basis.
I think it's unrealistic to try to get the candidate to solve real-world, day-to-day problems in an hour. This isn't what the job is like. "Design and implement this feature" is not a 45-minute task typically. It's typically days or weeks, so asking a "representative" problem in an interview is infeasible unless you're just doing high-level design, in which case it's not predictive for coding ability.
I could pick up a cookbook and study how to cook a souffle. I could learn enough about this academically that I could answer basically any questions you might ask. Having this academic knowledge is a good thing, but it doesn't mean I've ever even separated an egg. If you really want to know if I can make a souffle, your best bet is to hand me the stuff and ask me to do it.
> Here I disagree. If you have never used a framework or API, you don't know it. I don't care that you can regurgitate the Javadocs; I care that you know things like its pitfalls, quirks, and runtime oddities.
And here I assert that you're either probing on trivia or probing on things that can be learned without using the framework (actually, probably both). The pitfalls, quirks, and oddities are well documented in thousands of blogs.
> To be clear, I abhor trivia-based interviewing. What I am talking about is practical engineering tradeoffs between one course of action versus another, preferably supported by direct personal experience in the past.*
So this is an entirely different thing than pitfalls. Now you're talking about designing systems and dealing with tradeoffs. This isn't coding, and I don't believe you can always discover coding gaps by probing on this.
> Exposing the level of competence a five-minute coding problem can is trivial to do in parallel with a deep engineering discussion. In fact, I think some sort of code should be part of that discussion.
I guess I'm confused about what you're asking in an interview, then. Are your candidates coding or not?
> It should be something two-way, more representative of what the job is like on a daily basis.
I think it's unrealistic to try to get the candidate to solve real-world, day-to-day problems in an hour. This isn't what the job is like. "Design and implement this feature" is not a 45-minute task typically. It's typically days or weeks, so asking a "representative" problem in an interview is infeasible unless you're just doing high-level design, in which case it's not predictive for coding ability.
I think I'm not being clear and may be misunderstanding you.
When I was writing signal processing code, we had four levels of engineering.
- The first was an Algorithm Description Document. This document laid out, in mathematical terms, the algorithms used for various signal processing functions in the system. It was purely conceptual.
- The second was an Algorithm Implentation Document. This document mapped the algorithms to specific parts of the hardware and software system, laying out the logical module structure and data flow. It also specified how the algorithms would be realized in code, since parts of a particular algorithm might need to run in different parts of the system. The AID was still primarily mathematical.
- The third was a series of software design documents. These documents specified the details of module interfaces, code and file layouts, and specifics about what algorithms would be implemented in what functions.
- The fourth was the actual translation of the mathematical algorithms in the AID into actual C code. Most of the engineering at this level dealt with function-level optimization and some platform-specific stuff.
Of those four levels, only the last is what I would consider "coding". It is also the least important. Anyone who can really understand the first three levels and is willing to learn can handle the fourth. As I mentioned previously, there will be differences in efficiency, but not in capability. It would take willful ignorance for this not to be the case. I think this is also what you are considering "coding", but I'm not sure; I also think you might be merging #3 and #4.
Now, most commercial projects do not run this way and, as far as I know, few have dedicated systems engineers to manage the system architecture and APIs. Thus, the software engineers building the system generally perform the work in all four of these levels simultaneously as the system builds out. Despite this, I still only consider the work that fits level 4 to be "coding", and still consider it to be the least important.
What I look for when I interview are engineers who operate very well in levels 1 - 3. If I find that, level 4 is pretty much a given absent the rare pathological case or someone with a very "academic" attitude. The questions I personally ask deal with levels 2 and 3. Those questions cover some of what might be considered coding, like API design and class layout. I do not have candidates actually generate code, though. Some of the other interviewers have candidates write small functions in pseudocode or explain existing, uncommented Java code, though.
I also need to note that we have a take-home test that candidates need to pass before they get an on-site.We have also recently added an open coding challenge which may eventually replace the take-home test. This probably covers your requirement for candidates to code, even though it isn't done in front of one of us.
When I was writing signal processing code, we had four levels of engineering.
- The first was an Algorithm Description Document. This document laid out, in mathematical terms, the algorithms used for various signal processing functions in the system. It was purely conceptual.
- The second was an Algorithm Implentation Document. This document mapped the algorithms to specific parts of the hardware and software system, laying out the logical module structure and data flow. It also specified how the algorithms would be realized in code, since parts of a particular algorithm might need to run in different parts of the system. The AID was still primarily mathematical.
- The third was a series of software design documents. These documents specified the details of module interfaces, code and file layouts, and specifics about what algorithms would be implemented in what functions.
- The fourth was the actual translation of the mathematical algorithms in the AID into actual C code. Most of the engineering at this level dealt with function-level optimization and some platform-specific stuff.
Of those four levels, only the last is what I would consider "coding". It is also the least important. Anyone who can really understand the first three levels and is willing to learn can handle the fourth. As I mentioned previously, there will be differences in efficiency, but not in capability. It would take willful ignorance for this not to be the case. I think this is also what you are considering "coding", but I'm not sure; I also think you might be merging #3 and #4.
Now, most commercial projects do not run this way and, as far as I know, few have dedicated systems engineers to manage the system architecture and APIs. Thus, the software engineers building the system generally perform the work in all four of these levels simultaneously as the system builds out. Despite this, I still only consider the work that fits level 4 to be "coding", and still consider it to be the least important.
What I look for when I interview are engineers who operate very well in levels 1 - 3. If I find that, level 4 is pretty much a given absent the rare pathological case or someone with a very "academic" attitude. The questions I personally ask deal with levels 2 and 3. Those questions cover some of what might be considered coding, like API design and class layout. I do not have candidates actually generate code, though. Some of the other interviewers have candidates write small functions in pseudocode or explain existing, uncommented Java code, though.
I also need to note that we have a take-home test that candidates need to pass before they get an on-site.We have also recently added an open coding challenge which may eventually replace the take-home test. This probably covers your requirement for candidates to code, even though it isn't done in front of one of us.
Yes, I'm generally looking for someone who can do all of those things (though I don't work in DSP, so it's not an exact mapping). I would say that at least part of #3 is coding. If you're actually defining code layout etc., you are basically coding. If you dig deep enough here, you can probably rule out the vast majority of people who can't do #4. I still think it's a more efficient use of time to just ask them to do a bit of coding, though. Then I have certainty on the question rather than an implicit answer.
I worked with a team of these before. You can rule our certain types of people with tech questions (i.e. never coded before, only coded hacked, etc.). You cannot rule out
* People who read a lot of blogs so know the lingo, but never code
* People who get the theory but can't actually implement something that works without bugs
* People that over-engineer simple things
* People who read a lot of blogs so know the lingo, but never code
* People who get the theory but can't actually implement something that works without bugs
* People that over-engineer simple things
> People who read a lot of blogs so know the lingo, but never code
I think this is where my definition of "talk like a professional" may be more strict. I'm interested in understanding, in opinions, in the actual engineering.
> People who get the theory but can't actually implement something that works without bugs
Having spent most of my career working with PhDs trying to design and implement signal processing and NLP algorithms, I have experienced a lot of that. Given the proper attitude, it is fairly easy to correct coding deficiencies if someone really understands what they are supposed to be doing.
> People that over-engineer simple things
The greater offenders are fairly easy to detect in a directed technical discussion. The lesser offenders are fairly easy to correct in code review, again given the proper attitude, as they are unlikely to be doing things that require major refactoring.
In short, I think I place much more emphasis on engineering as opposed to coding. My experience has been that any coding issues people with good engineering skills and an attitude worth hiring have are quickly and easily corrected. You cannot really understand the engineering of a system without being able to code it. You might not be as efficient as someone else, but you are capable.
I think this is where my definition of "talk like a professional" may be more strict. I'm interested in understanding, in opinions, in the actual engineering.
> People who get the theory but can't actually implement something that works without bugs
Having spent most of my career working with PhDs trying to design and implement signal processing and NLP algorithms, I have experienced a lot of that. Given the proper attitude, it is fairly easy to correct coding deficiencies if someone really understands what they are supposed to be doing.
> People that over-engineer simple things
The greater offenders are fairly easy to detect in a directed technical discussion. The lesser offenders are fairly easy to correct in code review, again given the proper attitude, as they are unlikely to be doing things that require major refactoring.
In short, I think I place much more emphasis on engineering as opposed to coding. My experience has been that any coding issues people with good engineering skills and an attitude worth hiring have are quickly and easily corrected. You cannot really understand the engineering of a system without being able to code it. You might not be as efficient as someone else, but you are capable.
Also:
* People who have been on a team long enough to learn how things should work but lack the skills to really do the coding.
I phone screened a guy fairly recent who had a very long background (~20 years) in a pretty security-conscious field. He did very well in open-ended design discussions and was able to explain his prior work very well. He couldn't code trivial binary tree operations, though. I can only assume that someone else on his team is carrying most of the coding burden. This guy might be great at a PM-type job (or not, I don't interview for those jobs), but not as a dev.
* People who have been on a team long enough to learn how things should work but lack the skills to really do the coding.
I phone screened a guy fairly recent who had a very long background (~20 years) in a pretty security-conscious field. He did very well in open-ended design discussions and was able to explain his prior work very well. He couldn't code trivial binary tree operations, though. I can only assume that someone else on his team is carrying most of the coding burden. This guy might be great at a PM-type job (or not, I don't interview for those jobs), but not as a dev.
I feel like there is some detail missing here. What constituted doing "very well" on the open-ended questions? How deep into the engineering did you get? Since you mentioned he "might be great at a PM-type job" it sounds like you didn't go very deep on int.
That one was a phone screen, so no it was not the deepest technical dive ever. But in fairness a 1-hour interview is rarely the deepest technical dive possible. I normally pose some big problem like "design a system that accomplishes X at high scale" and then drill into some area iteratively until we're discussing fairly specific details. At the high level I'm looking for the candidate to recognize tradeoffs between centralized/distributed designs, sanely choose the major components, deal with stateful and stateless components, recognize security tradeoffs, etc. As we drill in we might talk about data partitioning, georeplication, or high availability and implementation tradeoffs there. We might get down to the level of something like handling out-of-order change notifications. Rarely do I drill down to the level of file I/O or memory management strategies.
>Invest in a competent hiring manager who can see through applicant bull shit and identify talent within a reasonable range. Includes basic negotiation skills. And assume the rightful risk that is employing another human being.
Problem with that is it's hard to find one. It's gotta be someone with pretty good coding skills. This person would then have to share their time between either coding head-down or managing coders, either of which are time-consuming and intolerant of disruption.
One idea I had was you could have mutual feedback. A bunch of people would after a while have anonymous input on each other, giving you say 10 coder's impressions of a fellow coder. This wouldn't have to be quizzes, half hour coffees could do.
Problem with that is it's hard to find one. It's gotta be someone with pretty good coding skills. This person would then have to share their time between either coding head-down or managing coders, either of which are time-consuming and intolerant of disruption.
One idea I had was you could have mutual feedback. A bunch of people would after a while have anonymous input on each other, giving you say 10 coder's impressions of a fellow coder. This wouldn't have to be quizzes, half hour coffees could do.
I recently graduated from college and started looking for my first programming jobs. I ran into a ton of these "take home" interviews and they were some of the most stressful things I've ever done.
The first one, they asked me to solve an incredibly complex math problem that I had no idea about so I struggled with it for 5 or so hours before giving up. Probably good that they didn't give me the job; if they were expecting me to know the level of math they were requesting then I would have had a pretty bad time. So I guess the test worked in that regard.
The second one interview at a different company, they gave me more moderate questions after a phone interview- mostly questions testing knowledge of a specific language. When I asked how long it would take, I was told "you have a deadline of 8 hours from when you download the zip file to when you turn in your code, expect it to take the entire 8 hours". That was probably the most stressful 8 hours of my life, way worse than any whtieboard interview would have been. I didn't look away from my computer once the whole time, did not let myself get up to eat, go to the bathroom, or anything. My girlfriend was in the room and tried to console me while I was doing it and stressing out and I ended up yelling at her to be quiet because I was trying to hard to concentrate. I finished the last problem about 10 minutes before the deadline, zipped the files up, and sent it to them. I never heard back.
Favorite interview I've ever had was a combo of a live coding session and a phone interview. They used an online tool that they gave me a link to that had syntax highlighting. The interviewer on the phone could see where I was typing and could highlight stuff etc. Think google docs, but for coding. They asked me to write some algorithms, then asked me to re-write them recursively, then asked a bunch of questions about the limitations of the algorithm, cases when it would be used, etc. It was fun, not stressful in any way, and the interviewer even taught me some things along the way! I feel like they got a really good picture of my thought processes and my personality so they could accurately judge if I would be a good fit on the team.
Paying somebody to do the take-home coding assignment and paying them would definitely help alleviate some of the stress, but I'm still not a fan of the format. You don't know how long it took the person to make their solutions, and you don't get a good idea for how they work, only what they can do (both are equally important, in my opinion).
The first one, they asked me to solve an incredibly complex math problem that I had no idea about so I struggled with it for 5 or so hours before giving up. Probably good that they didn't give me the job; if they were expecting me to know the level of math they were requesting then I would have had a pretty bad time. So I guess the test worked in that regard.
The second one interview at a different company, they gave me more moderate questions after a phone interview- mostly questions testing knowledge of a specific language. When I asked how long it would take, I was told "you have a deadline of 8 hours from when you download the zip file to when you turn in your code, expect it to take the entire 8 hours". That was probably the most stressful 8 hours of my life, way worse than any whtieboard interview would have been. I didn't look away from my computer once the whole time, did not let myself get up to eat, go to the bathroom, or anything. My girlfriend was in the room and tried to console me while I was doing it and stressing out and I ended up yelling at her to be quiet because I was trying to hard to concentrate. I finished the last problem about 10 minutes before the deadline, zipped the files up, and sent it to them. I never heard back.
Favorite interview I've ever had was a combo of a live coding session and a phone interview. They used an online tool that they gave me a link to that had syntax highlighting. The interviewer on the phone could see where I was typing and could highlight stuff etc. Think google docs, but for coding. They asked me to write some algorithms, then asked me to re-write them recursively, then asked a bunch of questions about the limitations of the algorithm, cases when it would be used, etc. It was fun, not stressful in any way, and the interviewer even taught me some things along the way! I feel like they got a really good picture of my thought processes and my personality so they could accurately judge if I would be a good fit on the team.
Paying somebody to do the take-home coding assignment and paying them would definitely help alleviate some of the stress, but I'm still not a fan of the format. You don't know how long it took the person to make their solutions, and you don't get a good idea for how they work, only what they can do (both are equally important, in my opinion).
Wow, that's pretty awful. I might posit some rules for good take-home tests:
1. A very loose time limit. Maybe, as a rule of thumb, at least 10x the time you expect the challenge to take (so, a 1-hour challenge is given a 10-hour limit to turn it in, and a 5-hour challenge is given two days). If you expect it to be done in real time, why they heck don't you just bring the candidate in for a normal interview?
2. If the challenge is expected to take more than, say, 5 hours, the candidate should be paid for their time. Long challenges are going to disincentive candidates with lots of good alternatives to your company, so compensating them as in your own interest if you want good engineers to make it through your process. Plus, it's a strong signal that you value their time.
3. The challenge should be as close as possible to the sort of thing you do day-to-day. This should be obvious, but don't ask math questions or graph traversal questions unless this sort of thing comes up regularly in the job you're hiring for.
4. The candidate should be given an opportunity to discuss their solution. In real-life code, a seemingly stupid decision is may well be totally justifiable, but not obviously so from the code itself.
5. The candidate always gets an explanation of why they were rejected. At one company I worked at, I was in the position of reviewing takehome tests (which were one component of a longer process which involved some on-site coding). I had to write those explanations, and I know it's hard. It's easy to say "your skills are not a good fit at this time", but it's hard to tell someone you don't know what's wrong with something they put a lot of work into. It's still worth doing.
1. A very loose time limit. Maybe, as a rule of thumb, at least 10x the time you expect the challenge to take (so, a 1-hour challenge is given a 10-hour limit to turn it in, and a 5-hour challenge is given two days). If you expect it to be done in real time, why they heck don't you just bring the candidate in for a normal interview?
2. If the challenge is expected to take more than, say, 5 hours, the candidate should be paid for their time. Long challenges are going to disincentive candidates with lots of good alternatives to your company, so compensating them as in your own interest if you want good engineers to make it through your process. Plus, it's a strong signal that you value their time.
3. The challenge should be as close as possible to the sort of thing you do day-to-day. This should be obvious, but don't ask math questions or graph traversal questions unless this sort of thing comes up regularly in the job you're hiring for.
4. The candidate should be given an opportunity to discuss their solution. In real-life code, a seemingly stupid decision is may well be totally justifiable, but not obviously so from the code itself.
5. The candidate always gets an explanation of why they were rejected. At one company I worked at, I was in the position of reviewing takehome tests (which were one component of a longer process which involved some on-site coding). I had to write those explanations, and I know it's hard. It's easy to say "your skills are not a good fit at this time", but it's hard to tell someone you don't know what's wrong with something they put a lot of work into. It's still worth doing.
You did an 8 hour test for them at full concentration, and they didn't even get back to you?
Please name these idiots.
Please name these idiots.
Not OP, but as someone who took a 6 hour test at full concentration, with no feedback other then a polite 'no', there are a few reasons I don't out them.
1) I didn't score well on the test. It's within reason they read this post and release my scores with a 'it was because he scored low'. Future employers then come across my scores and deny me.
2) Don't bite the hand that's like the hand that MIGHT feed you. You don't yell and storm out of investor meetings. Because then they tell other investors. Similar logic here.
3) This place is very optimistic. Pessimistic people aren't particularly desirable. Whistle blowers are included in the latter group especially by corporate entities. Who knows what it could cause.
It really boils down to the fact that it's not a safe environment to do so. Or at least it doesn't feel so.
1) I didn't score well on the test. It's within reason they read this post and release my scores with a 'it was because he scored low'. Future employers then come across my scores and deny me.
2) Don't bite the hand that's like the hand that MIGHT feed you. You don't yell and storm out of investor meetings. Because then they tell other investors. Similar logic here.
3) This place is very optimistic. Pessimistic people aren't particularly desirable. Whistle blowers are included in the latter group especially by corporate entities. Who knows what it could cause.
It really boils down to the fact that it's not a safe environment to do so. Or at least it doesn't feel so.
Getting back to you with a no (and nothing else) is quite a stretch from literally nothing as OP.
I think you're playing the game of life with your scared setting turned up too high. You might get better results to turn the bold up a bit over time as you gain confidence.
I think you're playing the game of life with your scared setting turned up too high. You might get better results to turn the bold up a bit over time as you gain confidence.
Just want to say, in conjecture to my sibling post on this thread, I had a identical experience. In fact, if you're in the Bay Area we likely applied to the same companies.
My take-homes made me fundamentally question my ability as a programmer. And I'm a damn good programmer. That is wrong on so many levels.
I can only pledge that if I'm ever in a position to hire developers (at any level) that I'll never force that kind of experience on a potential applicant.
I feel their is a major gap between junior and senior developers. And in this environment a junior developer has zero leverage with the companies he/she applies to. So they treat us like cattle.
It's like with every generation gap, their isn't much point blaming those who came before us. We can only do the best we can and make an effort not to replicate their mistakes. The next class(es) of software engineers have to improve and be better then the ones before them.
I'm preaching, but I hope you get what I mean.
My take-homes made me fundamentally question my ability as a programmer. And I'm a damn good programmer. That is wrong on so many levels.
I can only pledge that if I'm ever in a position to hire developers (at any level) that I'll never force that kind of experience on a potential applicant.
I feel their is a major gap between junior and senior developers. And in this environment a junior developer has zero leverage with the companies he/she applies to. So they treat us like cattle.
It's like with every generation gap, their isn't much point blaming those who came before us. We can only do the best we can and make an effort not to replicate their mistakes. The next class(es) of software engineers have to improve and be better then the ones before them.
I'm preaching, but I hope you get what I mean.
I can only say that if you're ever hiring, you probably will ask people to code for you. You won't at first, but after you've hired a few who could "talk the talk" but then can't actually write readable, maintainable, and in some cases working code, you'll reevaluate.
There's a gap between knowing concepts and knowing how to apply those concepts. There's actually a certain (limited) amount of artistry to writing code. Hiring a programmer without seeing code is like hiring an Architect without ever having seen a house he's designed.
There's a gap between knowing concepts and knowing how to apply those concepts. There's actually a certain (limited) amount of artistry to writing code. Hiring a programmer without seeing code is like hiring an Architect without ever having seen a house he's designed.
>I can only say that if you're ever hiring, you probably will ask people to code for you. You won't at first, but after you've hired a few who could "talk the talk" but then can't actually write readable, maintainable, and in some cases working code, you'll reevaluate.
You don't need an 8-hour take-home coding exercise to determine whether or not someone is capable of writing working code.
You don't need an 8-hour take-home coding exercise to determine whether or not someone is capable of writing working code.
Sorry to hear that. Do you remember what question or problem they asked you?
My two take-home experiences (one right out of college, one a couple years later) were fun, interesting, and much more laid-back than what you described. I would say if a company is placing such strict requirements on you and it is not a place you desperately want to work you should walk away. Hard for a recent college grad to do, but there are plenty of jobs out there.
Wow on the take homes. We have given taken homes that are just slightly more advanced than FizzBuzz. The problem would prompt the candidate to setup a basic build system (at the time in java) and then solve a problem using a couple basic patterns like observer and strategy. There was no real time limit other than we will not have you in for an in person interview until it is completed.
Like FizzBuzz I think the take home worked well as another data point. It easily filtered out people who couldn't follow basic instructions, people who were clueless about Java even though their resume said different, and people who simply didn't care. The problem also gave us something to talk about when the candidate came in for the in person interview.
Overall the problem was dead simple for anyone we would actually want to hire.
Like FizzBuzz I think the take home worked well as another data point. It easily filtered out people who couldn't follow basic instructions, people who were clueless about Java even though their resume said different, and people who simply didn't care. The problem also gave us something to talk about when the candidate came in for the in person interview.
Overall the problem was dead simple for anyone we would actually want to hire.
It can be said of most common interviewing techniques that they are very good at filtering out bad candidates, but very poor at picking out the really good ones.
I'm sorry that people are reluctant to name companies in these messages; I would absolutely love to know the companies in your first two examples.
Those sound like really bad take-home questions. I hope that better questions will improve the experience
I wonder if borrowing some experience from professors who use a take-home exam format might be a way to improve the quality. The college I went to (http://www.hmc.edu) heavily uses take-home exams, and it definitely seemed like a skill to write a good one. Not everything works well in that format, but it's possible to use it effectively.
> Offer $500 as a 1099 contractor to complete the assignment within the next 2 weeks or so.
Just as an aside, I would actually be more willing to undertake this type of interview if you weren't paying me. I don't have any ethical qualms about interviewing while working a full-time engineering job, but I'm not entirely comfortable taking a contracting job under the table. It may be something of a symbolic gesture given that it's such a small time investment and small amount of money, but it feels fundamentally dishonest, especially if you're a current or plausible future competitor to the company I currently work at.
The official policy where I currently work is that moonlighting is allowed but must be approved by our legal team, and while I could probably get away with not asking for permission, I'd be pretty uncomfortable in both cases (asking, or not asking).
Just as an aside, I would actually be more willing to undertake this type of interview if you weren't paying me. I don't have any ethical qualms about interviewing while working a full-time engineering job, but I'm not entirely comfortable taking a contracting job under the table. It may be something of a symbolic gesture given that it's such a small time investment and small amount of money, but it feels fundamentally dishonest, especially if you're a current or plausible future competitor to the company I currently work at.
The official policy where I currently work is that moonlighting is allowed but must be approved by our legal team, and while I could probably get away with not asking for permission, I'd be pretty uncomfortable in both cases (asking, or not asking).
Just give candidates the option to specify a charity of their choice to receive the $500 in their stead.
That is a good option too. Either defer it to a signing bonus or a charity of their choice.
> Just as an aside, I would actually be more willing to undertake this type of interview if you weren't paying me.
Sure, and I don't think I would force pay you. Would offer you the in person option, or the homework option without pay. Could maybe work it into a signing bonus if you took the job or something? Definitely not looking to put someone into an ethical dilemma, but not wanting to be one of those companies who makes candidates do really long take home interviews and then makes them mad when they don't extend an offer.
Sure, and I don't think I would force pay you. Would offer you the in person option, or the homework option without pay. Could maybe work it into a signing bonus if you took the job or something? Definitely not looking to put someone into an ethical dilemma, but not wanting to be one of those companies who makes candidates do really long take home interviews and then makes them mad when they don't extend an offer.
[deleted]
My guess is an applicant that stated the above would be allowed to work for free :)
Interesting and very ethical. Opposite of me. I don't see that a company should own your soul when you work for them. So doing a small amount of work for effectively beer money in your spare time is no problem. Even taking a second job is OK if it doesn't affect your first one.
Blanket prohibitions on moonlighting are illegal in California AFAICT, unless it's framed as a conflict-of-interest avoidance issue. See http://codes.lp.findlaw.com/cacode/LAB/1/d1/4/s96, section (k).
Just FYI, "under the table" means getting paid without proper documentation or withholding of taxes, FICA, etc. Doesn't really apply to 1099 situations.
Wouldn’t the solution be to ask the company to pay your favourite charity instead?
I like the idea of getting paid $500 or even $1k for a take home test. It's frankly a simple matter of respect. For employers, they're just asking you to do one task, but for employees, talking to 4-6 companies in an interview cycle can easily be 40 hours of interviewing and another 40 hours of work on top of that.
I'm at a point in my career where I mostly refuse take home tests because I'm busy and I value my free time. Getting paid would make me much more willing to jump through hoops. Particularly when that day of vacation cost me $350 after taxes, so it's not like I don't have skin in the game already.
I'm at a point in my career where I mostly refuse take home tests because I'm busy and I value my free time. Getting paid would make me much more willing to jump through hoops. Particularly when that day of vacation cost me $350 after taxes, so it's not like I don't have skin in the game already.
This was my thought as well, take home tests are a good test of your skills, but super annoying when you already have a job + family + hobbies.
>"Why should I program for free at home, my resume clearly shows I am already a skilled programmer. All this will do is cater to young people without families, or those fresh out of school".
I share this opinion, but only for certain ways of doing it. If it's done in a fully automated way, before even having a phone screen, then it's too easy to waste my time. As an alternative to the whiteboarding interview, however, it could be great.
I share this opinion, but only for certain ways of doing it. If it's done in a fully automated way, before even having a phone screen, then it's too easy to waste my time. As an alternative to the whiteboarding interview, however, it could be great.
> Offer $500 as a 1099 contractor to complete the assignment within the next 2 weeks or so.
I like this, not for the money, but because it guarantees that the company is not giving this assignment to 100 different candidates ($500 x 100 is getting prohibitive for a hiring budget). I'm more willing to put the effort if I know I'm one of the "finalists" for the position.
I like this, not for the money, but because it guarantees that the company is not giving this assignment to 100 different candidates ($500 x 100 is getting prohibitive for a hiring budget). I'm more willing to put the effort if I know I'm one of the "finalists" for the position.
Agreed that it's a positive thing to actually pay people for nontrivial amounts of work.
Downside is that, as a company paying someone to accomplish a task, I'd want to give a different task to each applicant, so that if they do a good job, it provides value to the company.
And this is a problem because rarely are two programming problems exactly as difficult as each other, so the test isn't normalized. Interviewing is hard enough when you keep all the variables that you can control the same.
Downside is that, as a company paying someone to accomplish a task, I'd want to give a different task to each applicant, so that if they do a good job, it provides value to the company.
And this is a problem because rarely are two programming problems exactly as difficult as each other, so the test isn't normalized. Interviewing is hard enough when you keep all the variables that you can control the same.
Make the number $250 and give everyone the same problem (which you have to change every so often because of glassdoor and the like).
That's less than you're already spending on salary for the interview slate, so you're at most doubling your already small costs for a critically important function for your company.
That's less than you're already spending on salary for the interview slate, so you're at most doubling your already small costs for a critically important function for your company.
The value to the company is making a good hire.
I mean, a bad hire is going to cost me thousands of dollars - if not tens of thousands of dollars. $500 is pretty cheap to figure that out up front on a candidate I am otherwise ready to hire. I am not going to do this to 100 people and pick the best, I am going to do it for someone that in every other way is a good to hire dev.
I don't think I would give out the same work to multiple people, let them put it into production on their first day and get the pride going ;)
I don't think I would give out the same work to multiple people, let them put it into production on their first day and get the pride going ;)
this^
before i produce work for free its good to know they are actually considering me and not just throwing me work as another 'step' in their process.
before i produce work for free its good to know they are actually considering me and not just throwing me work as another 'step' in their process.
$500 is too low - good developers will often charge $200+/hour freelancing, there is little incentive for them to deal with a take home test for that price with all of the time & stress that come with take home tests.
A choice is better than no choice though, but too many companies fumble through handling take home tests to make it worthwhile to a quality developer with any sense of value.
A choice is better than no choice though, but too many companies fumble through handling take home tests to make it worthwhile to a quality developer with any sense of value.
Yes, they charge $200+/h - when they actually do the work. This is my pet peeve lately when I see people throwing that number around. You're not actually working 24h. You likely don't work on the weekends. You take time to organise new work in between. Neither of the breaks are actually paid.
You get paid that much as a freelancer when you produce value for the company. So by doing an extra assignment, especially one that is a useless piece of code, no - $200/h is ridiculously high price.
You get paid that much as a freelancer when you produce value for the company. So by doing an extra assignment, especially one that is a useless piece of code, no - $200/h is ridiculously high price.
> Yes, they charge $200+/h - when they actually do the work.
Agreed.
Annual billable hours for consultants, be they developers, management, lawyers, et al, are considered huge if near 1500. That roughly works out to just under 30 billable hours per work week when vacations/holidays and non-billable time are taken into account.
As you surmise, just because someone _can_ bill X/hour doesn't mean that each hour they breathe commands the same remuneration.
Agreed.
Annual billable hours for consultants, be they developers, management, lawyers, et al, are considered huge if near 1500. That roughly works out to just under 30 billable hours per work week when vacations/holidays and non-billable time are taken into account.
As you surmise, just because someone _can_ bill X/hour doesn't mean that each hour they breathe commands the same remuneration.
But when comparing the difference, and considering a code project for an interview has a tendency to be even more stressful, which would a person rather spend their time on? It certainly is not the code project - I have heard & experienced too many stories where companies vastly underallocate time, or not respect that job seekers have other things that keep them busy typically, including less bureaucracy/time wasting (especially if a candidate has significant open source contributions), or not looking at the project, or even rejecting candidates for non-code related reasons (happened to someone I know today).
I'd much rather freelance for that type of stress & when having to compare the economics time-wise, which a take home test simply does not match - that is the whole point.
If a company does not want to pay that much, maybe they should respect the prospective employee's time, especially since it is a job seeker's market.
I'd much rather freelance for that type of stress & when having to compare the economics time-wise, which a take home test simply does not match - that is the whole point.
If a company does not want to pay that much, maybe they should respect the prospective employee's time, especially since it is a job seeker's market.
True but I am not hiring a freelancer. I don't think anyone spent a few hours interviewing to try and make $500 off me. I am just trying to be considered of their time, and make it clear I am not giving a giant programming assignment to 100 developers and only picking the 1 best, as that stinks for those doing the homework.
If you get to the paid homework step, we are ready to hire you, pending successful homework.
If you get to the paid homework step, we are ready to hire you, pending successful homework.
> But then other people say "Why should I program for free at home, my resume clearly shows I am already a skilled programmer.
Yes, and if I copy that resume and put my 6 year old's name on it, then her resume clearly shows she is a skilled programmer.
Yes, and if I copy that resume and put my 6 year old's name on it, then her resume clearly shows she is a skilled programmer.
This sounds like a great alternative and certainly would work well for me. What I'm afraid of though, is that I'd pass your interview and the aforementioned YCombinator companies would again want to put me through the on-site, high-pressure whiteboard technical interview.
That's definitely something we've started addressing as we optimize the process of matching programmers with the right startups. We're starting to gather data on how the YC companies run their hiring processes (seeing the degree of variance has been fascinating) and we can use that to avoid exactly this happening.
As many other parts of an interview, I've always found the blackboard coding session extremely strange. When was the last time you coded in TextEdit with no docs around, no time to think, standing up, and being watched over the shoulder?
I have pseudo-coded on blackboards many times, which is really what a blackboard coding interview should be. If you get a ding for writing .foreach instead of .forEach, that seems a bit picky ;)
However I realize I may have a bias as I do ok on most blackboard coding interviews I have done.. maybe been stumped 1 time out of 15 or so in my life? Some of it is a skill, that the more you do the better you get at it, but being good at it does not make you good at actual coding.
However I realize I may have a bias as I do ok on most blackboard coding interviews I have done.. maybe been stumped 1 time out of 15 or so in my life? Some of it is a skill, that the more you do the better you get at it, but being good at it does not make you good at actual coding.
It's exactly like being good at academic test taking.
I'm very good at memorization (of facts, ideas/concepts, and procedures/algorithms) and that allowed me to do well in my younger years and in standardised testing, so I always just scoffed when I heard other people complain about how it was a poor proxy for knowledge / abilities.
Of course, now I'm terrible at pretty much all the standard ways of interviewing developers (...here's where you say I'm probably just a bad developer) and now I have all this past-due empathy for those I disregarded before.
I'm very good at memorization (of facts, ideas/concepts, and procedures/algorithms) and that allowed me to do well in my younger years and in standardised testing, so I always just scoffed when I heard other people complain about how it was a poor proxy for knowledge / abilities.
Of course, now I'm terrible at pretty much all the standard ways of interviewing developers (...here's where you say I'm probably just a bad developer) and now I have all this past-due empathy for those I disregarded before.
> I have pseudo-coded on blackboards many times, which is really what a blackboard coding interview should be. If you get a ding for writing .foreach instead of .forEach, that seems a bit picky ;)
In my experience, some interviewers, particularly Amazon interviewers, get really anal about putting correct code on a whiteboard, sheet of paper, or bare text document. I personally do not when giving interviews.
My bigger problem, and I seem to depart from the vocal majority on this, is that my preferred normal workflow does not use whiteboards for anything besides task lists and diagrams. In 13 years[1] as an engineer I have never written actual or pseudocode on a whiteboard outside of an interview. I rarely do it on paper. I have never worked with anyone who did this very much, either.
[1] Granted, four of those were as an EE, but I still had to deal with code from time to time
In my experience, some interviewers, particularly Amazon interviewers, get really anal about putting correct code on a whiteboard, sheet of paper, or bare text document. I personally do not when giving interviews.
My bigger problem, and I seem to depart from the vocal majority on this, is that my preferred normal workflow does not use whiteboards for anything besides task lists and diagrams. In 13 years[1] as an engineer I have never written actual or pseudocode on a whiteboard outside of an interview. I rarely do it on paper. I have never worked with anyone who did this very much, either.
[1] Granted, four of those were as an EE, but I still had to deal with code from time to time
> In my experience, some interviewers, particularly Amazon interviewers, get really anal about putting correct code on a whiteboard
I had that experience with an Amazon interviewer. The guy who told me he expected perfect code also gave me zero feedback as I worked (despite me literally asking if I was headed in the direction he expected) and spent the entire time staring at his laptop. He'd apparently forgotten that he was scheduled for an interview, and also his manners.
I had that experience with an Amazon interviewer. The guy who told me he expected perfect code also gave me zero feedback as I worked (despite me literally asking if I was headed in the direction he expected) and spent the entire time staring at his laptop. He'd apparently forgotten that he was scheduled for an interview, and also his manners.
I was asked recently by a big company to code some classes in Python, on paper. They then questioned my indentation, syntax, and case-sensitivity multiple times!
A valid question, but paper-code?
A valid question, but paper-code?
Hah that is pretty terrible. In fact, I don't think Python is a very good language to code on paper due to the indentation side of things. Maybe graph paper would help?
Maybe! Or just understand it isn't easy for anyone to indent things on paper haha.
Approximately weekly, when explaining something to another engineer.
As I mentioned in another post here, I don't think I have ever written code on a whiteboard when explaining something. Explaining code has always happened in front of someone's computer, in an actual editor or IDE. Where I currently work, we typically pair on really hairy code.
Why don't you give yourself time to think? Why don't you use documentation?
Edit: Oh, and how do you version control the white board?
Edit: Oh, and how do you version control the white board?
how do you version control the white board?
We have a magic white board that can print out a big piece of paper showing everything on it. We also take pictures of it (and lesser white boards) to record designs agreed. Those pictures generally get put into a formal design document that is reviewed, signed, approved and all the rest of it.
We have a magic white board that can print out a big piece of paper showing everything on it. We also take pictures of it (and lesser white boards) to record designs agreed. Those pictures generally get put into a formal design document that is reviewed, signed, approved and all the rest of it.
[deleted]
> Why don't you give yourself time to think?
Explanations work better without lengthy pauses. Ideally, I can explain the thing without stopping too long.
> Why don't you use documentation?
I do if it's necessary. Sometimes it's not necessary.
> Oh, and how do you version control the white board?
I don't.
Explanations work better without lengthy pauses. Ideally, I can explain the thing without stopping too long.
> Why don't you use documentation?
I do if it's necessary. Sometimes it's not necessary.
> Oh, and how do you version control the white board?
I don't.
You can snap a picture of a filled board with your cell phone.
And you put the picture in version control?
There should be a time stamp in the exif data, so you could track progression from one photo to the next as a very primitive form of version control.
My original comment was mostly meant as snark, but I've had an interviewer take photos during a whiteboard exercise for this reason.
My original comment was mostly meant as snark, but I've had an interviewer take photos during a whiteboard exercise for this reason.
Not often, but I've taken a handwritten diagram and converted it to paths in inkscape, andd cleaned it up a bit.
The resulting svg or png can be then added to the docs folder of a project. Works well with sphinx, etc.
The resulting svg or png can be then added to the docs folder of a project. Works well with sphinx, etc.
TextEdit would be an upgrade. Try inserting lines on a blackboard.
I really hope this works. I never understood whiteboard interviews. They don't reflect how real development works or how that developer works either. I hope you post an update when you have enough info!
> We expect them to spend about 3 hours on the project (or as long as they want to spend to show us that they're a good programmer).
3 hours isn't too bad. I had a friend interview with some hedge fund and he said he probably spent 30+ hours on a take-home project (he didn't get the job). I thought that was way too excessive for just 1 interview.
3 hours isn't too bad. I had a friend interview with some hedge fund and he said he probably spent 30+ hours on a take-home project (he didn't get the job). I thought that was way too excessive for just 1 interview.
Do this instead of white boarding or other coding during the interview. Please.
I like these great in concept. When I've been between jobs, I've been very happy to participate in them.
A couple open questions:
1 - Is it reasonable to expect a 10x programmer whom your are trying to poach to give up so much time? (Or should you give them a $250 Starbucks gift card or something similar for their time?)
2 - Can you really ferret out cheating? I had a grad school classmate who paid someone to do his (non-programming) take-home homework for a job interview, and he got the job. He only lasted 6 or 7 months, but it was enough to be awful for all parties involved. I don't have a great counter-solution other than ask for someone to come in to the office to do the work, and even then you can't tell if they have remote support.
A couple open questions:
1 - Is it reasonable to expect a 10x programmer whom your are trying to poach to give up so much time? (Or should you give them a $250 Starbucks gift card or something similar for their time?)
2 - Can you really ferret out cheating? I had a grad school classmate who paid someone to do his (non-programming) take-home homework for a job interview, and he got the job. He only lasted 6 or 7 months, but it was enough to be awful for all parties involved. I don't have a great counter-solution other than ask for someone to come in to the office to do the work, and even then you can't tell if they have remote support.
If you read the article, the second part of the interview is doing more work on top of the project, but in front of people. I've gone through interviews like that before. Do two steps on a three step kata in front on your own time, and then a 2/3 hour interview to talk about what you wrote, and add the last piece to the kata. Not very easy to fake.
Any experienced programmer that is considering a job change will spend more than a few hours on the process anyway, in due diligence. If anything, I think this is an important part of gaining experience in the field: You actually want more time around future employers, to avoid a bad fit. Going to a place just to quit in 3 months time is a waste of your own time. Investing an extra few hours, or even few days, makes a lot of sense.
Any experienced programmer that is considering a job change will spend more than a few hours on the process anyway, in due diligence. If anything, I think this is an important part of gaining experience in the field: You actually want more time around future employers, to avoid a bad fit. Going to a place just to quit in 3 months time is a waste of your own time. Investing an extra few hours, or even few days, makes a lot of sense.
Won't the same programmer who got flustered in front of a white board also get flustered explaining the code?
I'm being the devils advocate here, trying to hone in on the best way to do this. I still think work products are a much better predictor than most any other interviewing technique.
I'm being the devils advocate here, trying to hone in on the best way to do this. I still think work products are a much better predictor than most any other interviewing technique.
Not necessarily, they are quite different forms of communication.
Coming up with something that works, versus explaining why it works or why you did it that way after the fact.
I think it's the "thinking out loud" part before you have a solution, that can fluster a lot of introverts.
Coming up with something that works, versus explaining why it works or why you did it that way after the fact.
I think it's the "thinking out loud" part before you have a solution, that can fluster a lot of introverts.
for some it's not about introversion/extroversion but simply the fact that whatever part of their brain deals with speech is the same part that deals with thinking through those kinds of problems, so they can't do both at the same time as a matter of physics.
this is to put it precisely. the reality is much messier and varied. the point is, people's brains are not wired the same. and what this means, to make a long story short, is that these interviews are not comparing apples to oranges... not even close.
this is to put it precisely. the reality is much messier and varied. the point is, people's brains are not wired the same. and what this means, to make a long story short, is that these interviews are not comparing apples to oranges... not even close.
[edit]oops. obviously, i meant apples to apples :D
1 - We don't expect anyone to give up this much time. I realize that it's a big ask. However, a lot of programmers we've spoken too really want this option. Anyone who does not want to is totally free to go with the (faster) regular interview. Either they let is watch them code for a short time, or they code for a longer time on their own.
2 - I can't say for sure (we're only just launching this), but I hope (and think) that talking with someone for 45-minutes about the project that they did will be enough to make sure that they actually did it themselves
2 - I can't say for sure (we're only just launching this), but I hope (and think) that talking with someone for 45-minutes about the project that they did will be enough to make sure that they actually did it themselves
Out of curiosity, what are the demographics of the people who made these suggestions?
Were they employed? Were they very desirable to companies? Were they very undesirable to companies? Did they have financial hardships? College graduates? College dropouts?
I think it's important to remember how diverse the hiring pool is, and which voices tend to be the loudest. I would argue that the advice most companies get are from very successful companies. Similarly engineers likely seek advice from very successful engineers.
Both these groups are in the minority I think, and would thus not be in the best position to dictate for the majority (strictly in terms of fulfilling individual needs that they themselves have not encountered, personally or otherwise).
Example: 10x programmer wants a hiring process that caters to 10x programmers.
Problem: Vast majority of programmers do not fall into this category, and would not benefit from such a system. Yet companies want 10xers, so they go with it anyways. Then they complain about how hard it is to hire programmers.
Were they employed? Were they very desirable to companies? Were they very undesirable to companies? Did they have financial hardships? College graduates? College dropouts?
I think it's important to remember how diverse the hiring pool is, and which voices tend to be the loudest. I would argue that the advice most companies get are from very successful companies. Similarly engineers likely seek advice from very successful engineers.
Both these groups are in the minority I think, and would thus not be in the best position to dictate for the majority (strictly in terms of fulfilling individual needs that they themselves have not encountered, personally or otherwise).
Example: 10x programmer wants a hiring process that caters to 10x programmers.
Problem: Vast majority of programmers do not fall into this category, and would not benefit from such a system. Yet companies want 10xers, so they go with it anyways. Then they complain about how hard it is to hire programmers.
Less experienced programmers are definitely over represented in engineers adversely affected by stress. This could be because people get better at interviewing at they gain experience, or because the people who can't perform under interview stress drop out of the profession
> 2 - Can you really ferret out cheating? I had a grad school classmate who paid someone to do his (non-programming) take-home homework for a job interview, and he got the job. He only lasted 6 or 7 months, but it was enough to be awful for all parties involved. I don't have a great counter-solution other than ask for someone to come in to the office to do the work, and even then you can't tell if they have remote support.
Personally, I think this is where the skill of the interviewer comes in. A skilled interviewer can bust through bullshit fairly quickly. A big problem in interviewing and hiring right now is that most of the interviewers are not skilled. Thus, companies try to come up with processes and metrics that (theoretically) remove the interviewer skill from the equation. Unfortunately, this seems to be done in a horribly unscientific manner.
Personally, I think this is where the skill of the interviewer comes in. A skilled interviewer can bust through bullshit fairly quickly. A big problem in interviewing and hiring right now is that most of the interviewers are not skilled. Thus, companies try to come up with processes and metrics that (theoretically) remove the interviewer skill from the equation. Unfortunately, this seems to be done in a horribly unscientific manner.
If only we had some sort of take-home exercise for interviewers.
I hear you on unskilled interviews.
I guess the challenge in this situation is that the whole reason for take-home work is that introverted interviewees get flustered in person. Won't this happen when the review happens?
This still seems like a better idea than "Tell me about yourself" and "How many golf balls fit in a 747?"
I guess the challenge in this situation is that the whole reason for take-home work is that introverted interviewees get flustered in person. Won't this happen when the review happens?
This still seems like a better idea than "Tell me about yourself" and "How many golf balls fit in a 747?"
"Won't this happen when the review happens?"
At that point, the candidate should not be hired.
If the candidate can't communicate while writing code, and can't communicate about code already written, at some point you have to be afraid the candidate just can't communicate.
If this is a person who can communicate in any situation except an interview situation, then it's a Catch 22 where their ability to communicate in a work environment just can't be verified.
(Unless you secretly record them communicating in some other work context? Probably not a scalable approach.)
At that point, the candidate should not be hired.
If the candidate can't communicate while writing code, and can't communicate about code already written, at some point you have to be afraid the candidate just can't communicate.
If this is a person who can communicate in any situation except an interview situation, then it's a Catch 22 where their ability to communicate in a work environment just can't be verified.
(Unless you secretly record them communicating in some other work context? Probably not a scalable approach.)
I like this idea. I took the Triplebyte quiz mostly out of curiosity and then decided to stop when I was prompted for a phone interview. I like the idea of "pre-qualifying" myself for a job even before I'm ready, so that when the time is right, I can pull the trigger and switch easily.
With a phone interview you have to commit, which is no problem if you're in the market. But if you're not in the market and you still want your options to be open, interviewing on your own time makes a lot of sense and reduces the mental burden and stress of switching.
With a phone interview you have to commit, which is no problem if you're in the market. But if you're not in the market and you still want your options to be open, interviewing on your own time makes a lot of sense and reduces the mental burden and stress of switching.
One thing that a local company did was to give a simple programming assignment (given postgres database connection info, create simple REST API using any language). They would switch up the details of the assignment for each candidate (e.g. different endpoints for same set of tables, different tables, etc.) to combat cheating.
The applicant was expected to be able to answer questions about the code anyways, so hopefully that would've been an effective layer of protection as well. Don't know if this company is still doing it.
The applicant was expected to be able to answer questions about the code anyways, so hopefully that would've been an effective layer of protection as well. Don't know if this company is still doing it.
How does this work if candidate has experience in, say, backend, but haven't worked much on frontends or mobile apps etc? A specific "take home" task puts those candidates at advantage who have already worked in related area, have written code that they can immediately reuse or are experienced in frameworks that allows them to fast trek. This would be great if the job also required exact same capabilities for the foreseeable future from the candidate. However it would overrate or underrate the candidate if the job requires candidates to work on diverse set of problems in long run (for example, switching from MySQL to graph problems). It would be interesting if article also gave few examples of such "take home" tests.
Take home interviews are common in data science, and they are deeply exploitative. I have been asked to spend anywhere between an hour to a week on a project, and in most cases they simply decline to hire you without giving you any feedback. In the worst cases, companies like Knewton and Mattermark have given take home interviews that not only took a lot of time, but were related to the companies core business model. In other words, it's free consulting.
And finally, you aren't fooling anyone when you say that it should take 3 hours, or as long as they want to spend on it, while giving them 3 days. You are pressuring people to spend as much time on the problem as possible to "prove they are a good programmer." It's exploitative - you aren't paying us and the probability that we will get hired is extremely small. Think about how many companies we are interviewing with.
And finally, you aren't fooling anyone when you say that it should take 3 hours, or as long as they want to spend on it, while giving them 3 days. You are pressuring people to spend as much time on the problem as possible to "prove they are a good programmer." It's exploitative - you aren't paying us and the probability that we will get hired is extremely small. Think about how many companies we are interviewing with.
Gauging ability to operate under pressure is valuable too...ever tried to fix a bug on a device that was supposed to ship yesterday, under the pressure of the entirety of the hierarchy above you all the way up to the CxO?
It helps to know if the candidate you hire will handle that, or fall apart just when you need him/her to do the job he/she is paid for.
It helps to know if the candidate you hire will handle that, or fall apart just when you need him/her to do the job he/she is paid for.
Um, you really can't interview for that person, thanks. That person is someone who is already in your company and gets promoted to "trusted lieutenant". Promotion to "completely trusted lieutenant" is only awarded posthumously. :)
If that kind of situation is happening more than once in a blue moon, something is very wrong at your company.
If that kind of situation is happening more than once in a blue moon, something is very wrong at your company.
As an organisation you have to decide on what you want. The developers who can handle high stress situations are rarely the most skilled developers.
If you are hiring for a high stress environment then you need to focus upon candidates who can handle stressful environments over candidates who write really good software.
On the other hand most environments are not high stress.
If you are hiring for a high stress environment then you need to focus upon candidates who can handle stressful environments over candidates who write really good software.
On the other hand most environments are not high stress.
For what it's worth, the author included this footnote about what you mentioned.
"1. The stress of interviewing seems to be different than the stress of performing a job. None of the people we've spoken to who do poorly in interviews report problems performing under deadlines at work, or when a website is down and there's pressure to get it back up."
"1. The stress of interviewing seems to be different than the stress of performing a job. None of the people we've spoken to who do poorly in interviews report problems performing under deadlines at work, or when a website is down and there's pressure to get it back up."
Rather than pick a new project, however, they'll take the same project further, incorporating feedback from the 1st interview.
Software developers huh! What other role requires you to complete an exam to be considered for a job (or even just an interview). Moreover a 4-8 hour exam with no syllabus. No 'past papers' etc.
I think this should stop. In it's place, developers can have a pet open-source project, which they submit with their application. The point is that the same project can be submitted for a hundred applications if needed, saving the candidate dozens or hundreds of hours of wasted time.
They could put that time into the project instead and even if they don't get a single offer, the open source community benefits, and the candidate got to do something interesting rather than convert roman numerals or aggregate an array.
I seriously wonder if we should all boycott dev tests. If they give a test, just give them a github link and say "Please review this as a good example of my work.". If they can't review code that isn't tied to a silly question maybe they are not worth working for.
I say this having done some such tests recently myself. (To be fair one of them said a Github submission is OK, I just didn't feel my Github was good enough at the time.). I kind of feel bad contributing to the madness!
I think this should stop. In it's place, developers can have a pet open-source project, which they submit with their application. The point is that the same project can be submitted for a hundred applications if needed, saving the candidate dozens or hundreds of hours of wasted time.
They could put that time into the project instead and even if they don't get a single offer, the open source community benefits, and the candidate got to do something interesting rather than convert roman numerals or aggregate an array.
I seriously wonder if we should all boycott dev tests. If they give a test, just give them a github link and say "Please review this as a good example of my work.". If they can't review code that isn't tied to a silly question maybe they are not worth working for.
I say this having done some such tests recently myself. (To be fair one of them said a Github submission is OK, I just didn't feel my Github was good enough at the time.). I kind of feel bad contributing to the madness!
> What other role requires you to complete an exam to be considered for a job (or even just an interview). Moreover a 4-8 hour exam with no syllabus. No 'past papers' etc
As far as I can tell, essentially all of them. Of course, many of them have no practical way to even attempt to assess competence, so they fall back on "situational" and "culture fit" bullshit.
> I think this should stop. In it's place, developers can have a pet open-source project, which they submit with their application.
Problem the 1st: how do you verify the example project is actually the candidate's own work?
Problem the 2nd: building and maintaining a non-trivial open source project is a huge investment of time. There are many competent programmers who have a long list of things they must / would rather do when they get home from sitting at a computer programming all day. It's only more efficient if you send out 100s of applications, but as far as I can tell, competent programmers rarely spam 100s of applications. If you're just putting your toe in the water from time to time, taking a day off to do a traditional tech interview grill fest is a much more effective use of your time.
Problem the 3rd: evaluating a large, unfamiliar codebase is actually really hard. All unfamiliar code looks shitty, because you don't have the full context to explain the choices made. Very few people have GitHub pages that actually look good.
> I seriously wonder if we should all boycott dev tests.
Good luck with that co-ordination problem.
Tech interviews suck, for everyone involved. I still haven't heard a proposal for an alternative that isn't fatally flawed. (I consider "take home interviews" to be an incremental improvement on the basic whiteboard problem-solving test)
As far as I can tell, essentially all of them. Of course, many of them have no practical way to even attempt to assess competence, so they fall back on "situational" and "culture fit" bullshit.
> I think this should stop. In it's place, developers can have a pet open-source project, which they submit with their application.
Problem the 1st: how do you verify the example project is actually the candidate's own work?
Problem the 2nd: building and maintaining a non-trivial open source project is a huge investment of time. There are many competent programmers who have a long list of things they must / would rather do when they get home from sitting at a computer programming all day. It's only more efficient if you send out 100s of applications, but as far as I can tell, competent programmers rarely spam 100s of applications. If you're just putting your toe in the water from time to time, taking a day off to do a traditional tech interview grill fest is a much more effective use of your time.
Problem the 3rd: evaluating a large, unfamiliar codebase is actually really hard. All unfamiliar code looks shitty, because you don't have the full context to explain the choices made. Very few people have GitHub pages that actually look good.
> I seriously wonder if we should all boycott dev tests.
Good luck with that co-ordination problem.
Tech interviews suck, for everyone involved. I still haven't heard a proposal for an alternative that isn't fatally flawed. (I consider "take home interviews" to be an incremental improvement on the basic whiteboard problem-solving test)
> Problem the 1st: how do you verify the example project is actually the candidate's own work?
Get them to talk about it in the interview
> Problem the 2nd: building and maintaining a non-trivial open source project is a huge investment of time...
Not really - in one weekend you probably can create something interesting and good enough for showing off to an employer. Also it will probably draw you in and you spend more time (for the fun of it) making it better. But you don't have to.
For those people who prefer FizzBuzz tests to implementing interesting stuff, I suggest companies provide the option to do a test for those who don't want to submit their project code.
> Good luck with that co-ordination problem.
A first stage could be that we say "no tests until at least an interview has occurred and the candidate has progressed to the next stage". That is in any candidates own interests regardless of other's actions - so wouldn't require coordination, just communication of the idea. It is in his own interests because the time spent on the speculative work could be better yielded doing something else, such as attending a meetup and generating more opportunities. Looking at it from a sales perspective of how to spend your time - lead qualification, lead generation etc. Why spend a lot of effort on an unqualified lead?
Get them to talk about it in the interview
> Problem the 2nd: building and maintaining a non-trivial open source project is a huge investment of time...
Not really - in one weekend you probably can create something interesting and good enough for showing off to an employer. Also it will probably draw you in and you spend more time (for the fun of it) making it better. But you don't have to.
For those people who prefer FizzBuzz tests to implementing interesting stuff, I suggest companies provide the option to do a test for those who don't want to submit their project code.
> Good luck with that co-ordination problem.
A first stage could be that we say "no tests until at least an interview has occurred and the candidate has progressed to the next stage". That is in any candidates own interests regardless of other's actions - so wouldn't require coordination, just communication of the idea. It is in his own interests because the time spent on the speculative work could be better yielded doing something else, such as attending a meetup and generating more opportunities. Looking at it from a sales perspective of how to spend your time - lead qualification, lead generation etc. Why spend a lot of effort on an unqualified lead?
Even trivial code is fine on Github. All I look for is that someone can actually write a bit of code, and a GH repo helps me quickly asses otherwise I have to do more digging. I've interviewed entirely too many people who are clueless and leave me wondering how they have ever had a job writing software.
> For those people who prefer FizzBuzz tests to implementing interesting stuff, I suggest companies provide the option to do a test for those who don't want to submit their project code.
In fact a great personal GH repo could be to implement FizzBuzz in a bunch of different languages. I think most companies would get an appreciate the joke.
> For those people who prefer FizzBuzz tests to implementing interesting stuff, I suggest companies provide the option to do a test for those who don't want to submit their project code.
In fact a great personal GH repo could be to implement FizzBuzz in a bunch of different languages. I think most companies would get an appreciate the joke.
"Problem the 1st: how do you verify the example project is actually the candidate's own work?"
Have them discuss the code in depth. Architectural and design decisions, coding conventions, alternate approaches they rejected, etc. Pretty hard to fake this if they didn't write the code.
"Problem the 3rd: evaluating a large, unfamiliar codebase is actually really hard."
Good chance to assess the candidate's ability to document. :)
More seriously, though, this is again an opportunity for the candidate to walk you through the code.
Have them discuss the code in depth. Architectural and design decisions, coding conventions, alternate approaches they rejected, etc. Pretty hard to fake this if they didn't write the code.
"Problem the 3rd: evaluating a large, unfamiliar codebase is actually really hard."
Good chance to assess the candidate's ability to document. :)
More seriously, though, this is again an opportunity for the candidate to walk you through the code.
Yes, but that switches the cost back onto the company instead of the employee.
Software devs have extremely low social status, despite pretty good pay.
Honestly I'm glad they don't watch "Survivor" and make us eat bugs or do weird obstacle courses. Seriously. You treat people like dirt, they're gonna resent it and complain on HN and some Stockholm Syndrome victims will back up the abusers.
Another example of the low social status is those horrible open offices, like a call center. Ugh.
Another example of immensely low social status is crazy work hours.
Honestly I'm glad they don't watch "Survivor" and make us eat bugs or do weird obstacle courses. Seriously. You treat people like dirt, they're gonna resent it and complain on HN and some Stockholm Syndrome victims will back up the abusers.
Another example of the low social status is those horrible open offices, like a call center. Ugh.
Another example of immensely low social status is crazy work hours.
> Another example of the low social status is those horrible open offices, like a call center. Ugh.
So, where you work software devs are in an open-plan space and other roles have their own office?
So, where you work software devs are in an open-plan space and other roles have their own office?
[deleted]
Many other fields require competency tests. What other field requires you to do a potentially unbounded amount of free work in the open and give it away to everybody?
Not everybody has any major projects on Github. The lack thereof does not signal lack of programming aptitude. If somebody has done a lot of great work, but it was all for employers and clients, that doesn't mean they would do poor work for you if you were their employer or client. I find that suggestion much more distasteful than having relevant work-sample tests.
Not everybody has any major projects on Github. The lack thereof does not signal lack of programming aptitude. If somebody has done a lot of great work, but it was all for employers and clients, that doesn't mean they would do poor work for you if you were their employer or client. I find that suggestion much more distasteful than having relevant work-sample tests.
Licensing should be up to the candidate, and it could be commercially licensed and not available to everybody.
You shouldn't have to write a OS Kernel or a modern browser in your spare time either. I'd say it would be about 4-8 hours work that shows off your skills.
But if you prefer to spend that same time on some inane made-up problem for one employer and code something that you will wont enjoy writing and never use again, then as I said in another reply that there should be the option for that too.
> Many other fields require competency tests
Is there a field that requires 4-8hr competency tests where each test is wildly different from the previous, with completely opaque criteria. At least with the provide your own project idea, if the employer has fucked up criteria the candidate has only wasted a few seconds of their time in sending the zip file.
Kind of analogous to 'show me your portfolio' for a photographer. The portfolio should be enough. Unfortunately this is not possible as a developer as the source from your previous jobs is likely to be closed, hence the need for a personal project.
You shouldn't have to write a OS Kernel or a modern browser in your spare time either. I'd say it would be about 4-8 hours work that shows off your skills.
But if you prefer to spend that same time on some inane made-up problem for one employer and code something that you will wont enjoy writing and never use again, then as I said in another reply that there should be the option for that too.
> Many other fields require competency tests
Is there a field that requires 4-8hr competency tests where each test is wildly different from the previous, with completely opaque criteria. At least with the provide your own project idea, if the employer has fucked up criteria the candidate has only wasted a few seconds of their time in sending the zip file.
Kind of analogous to 'show me your portfolio' for a photographer. The portfolio should be enough. Unfortunately this is not possible as a developer as the source from your previous jobs is likely to be closed, hence the need for a personal project.
> Licensing should be up to the candidate, and it could be commercially licensed and not available to everybody.
> You shouldn't have to write a OS Kernel or a modern browser in your spare time either. I'd say it would be about 4-8 hours work that shows off your skills.
Well, this is a different suggestion. Now we're only talking about a few hours of work rather than a pet project. I don't think the licensing makes much of a difference since I probably couldn't monetize four hours of randomly dorking around. I'm still not a huge fan of the idea, though, because it sounds hard to me to show off (adequately) skills I've spent decades acquiring with less than a day's worth of aimless work. I don't feel like it would be very impressive.
> Is there a field that requires 4-8hr competency tests where each test is wildly different from the previous, with completely opaque criteria.
Nobody is advocating for eight-hour tests. The OP said they expected their test to take less than half that. And most fields that require competency tests (e.g. cooking, acting, music) will expect you to do it live, on the spot. Take-home tests are being suggested in the case of programming specifically because programmers hate being expected to perform on the spot.
> At least with the provide your own project idea, if the employer has fucked up criteria the candidate has only wasted a few seconds of their time in sending the zip file.
Not all criteria that someone might test for are "fucked up." If I'm looking for, say, somebody to do a math-heavy machine learning job, and your pet open-source project is a jQuery plugin that makes web page backgrounds sparkly, the mismatch between those two is not the fault of my criteria.
> You shouldn't have to write a OS Kernel or a modern browser in your spare time either. I'd say it would be about 4-8 hours work that shows off your skills.
Well, this is a different suggestion. Now we're only talking about a few hours of work rather than a pet project. I don't think the licensing makes much of a difference since I probably couldn't monetize four hours of randomly dorking around. I'm still not a huge fan of the idea, though, because it sounds hard to me to show off (adequately) skills I've spent decades acquiring with less than a day's worth of aimless work. I don't feel like it would be very impressive.
> Is there a field that requires 4-8hr competency tests where each test is wildly different from the previous, with completely opaque criteria.
Nobody is advocating for eight-hour tests. The OP said they expected their test to take less than half that. And most fields that require competency tests (e.g. cooking, acting, music) will expect you to do it live, on the spot. Take-home tests are being suggested in the case of programming specifically because programmers hate being expected to perform on the spot.
> At least with the provide your own project idea, if the employer has fucked up criteria the candidate has only wasted a few seconds of their time in sending the zip file.
Not all criteria that someone might test for are "fucked up." If I'm looking for, say, somebody to do a math-heavy machine learning job, and your pet open-source project is a jQuery plugin that makes web page backgrounds sparkly, the mismatch between those two is not the fault of my criteria.
> because it sounds hard to me to show off (adequately) skills I've spent decades acquiring with less than a day's worth of aimless work
So you would be against a developer test too then?
> (e.g. cooking, acting, music) will expect you to do it live, on the spot.
Fair enough. So the developer equivalent should be a 1 hour technical interview, with perhaps a coding exercise on a computer within that hour. I am happy with those. To reduce the pressure the interviewer could leave the room for 10 mins while you code something. It is the do 4 hours work in your own time for a chance to get an interview or get to the next round that bugs me. And in reality I'd spend 8 just to maximize the chance of getting the job.
> your pet open-source project is a jQuery plugin that makes web page backgrounds sparkly
Yes there are sub divisions in the 'software developer' category. If you want a machine learning job you kind of have to demonstrate it one way or another. I argue that the option to show your existing code rather than do additional work on a made up problem is valuable.
So you would be against a developer test too then?
> (e.g. cooking, acting, music) will expect you to do it live, on the spot.
Fair enough. So the developer equivalent should be a 1 hour technical interview, with perhaps a coding exercise on a computer within that hour. I am happy with those. To reduce the pressure the interviewer could leave the room for 10 mins while you code something. It is the do 4 hours work in your own time for a chance to get an interview or get to the next round that bugs me. And in reality I'd spend 8 just to maximize the chance of getting the job.
> your pet open-source project is a jQuery plugin that makes web page backgrounds sparkly
Yes there are sub divisions in the 'software developer' category. If you want a machine learning job you kind of have to demonstrate it one way or another. I argue that the option to show your existing code rather than do additional work on a made up problem is valuable.
Sounds good in theory. But if enough companies started doing this, I can guarantee you that Indian companies will spring that will offer to create an open source project for you for $100 to $1000 depending on complexity of project, and amount of activity the github profile would show.
I wish I was kidding.
Same could be said of a take home test. You can outsource it for $100. I doubt it would get you anywhere at a company worth it's salt.
While that is sort of true for a take-home test, there is still value in giving candidates an online or in-person programming task that needs to be completed in a small amount of time (e.g. 1 hour)
FWIW most of the other high-payed jobs have some kind of entrance barrier too, usually in the form of having absolved a Univerity-level course and passed the exams.
Anybody can legally claim to be a programmer, so it makes sense for the companies to do some basic checking that a University would have done for them in other cases.
Anybody can legally claim to be a programmer, so it makes sense for the companies to do some basic checking that a University would have done for them in other cases.
> What other role requires you to complete an exam to be considered for a job (or even just an interview).
I know of a place in Chicago that does that for a dev job.
I know of a place in Chicago that does that for a dev job.
I thought Gayle Laakman McDowell, who wrote "Cracking the Coding Interview", wrote a good article about the problems of take-home interviews.
http://www.gayle.com/blog/2013/09/18/companies-who-give-cand...
At the core of the problem is that this approach can be used to burn a lot of a candidate's time without an equivalent investment from the company.
I've mentioned this before - I applied (maybe 5 years ago) to a company that first asked me to take a Java test (about 1 hour of work). Then the recruiter called me and sent me a take home project that should take "5-7 hours". I did this, crickets chirped for a month, though I did check in with the recruiter for a while (I gave up eventually). Finally the recruiter called me with a one-line brush off "we've decided not to move forward at this time…"
Granted, this was a bad experience, and it won't always be like this. But I'm pretty close to saying "never" to these exams.
I thought Gayle's suggestions were good. She goes so far as to suggest a 90% passage rate - that you should not be giving these tests to people who are unlikely to pass then, using them to confirm, not screen.
http://www.gayle.com/blog/2013/09/18/companies-who-give-cand...
At the core of the problem is that this approach can be used to burn a lot of a candidate's time without an equivalent investment from the company.
I've mentioned this before - I applied (maybe 5 years ago) to a company that first asked me to take a Java test (about 1 hour of work). Then the recruiter called me and sent me a take home project that should take "5-7 hours". I did this, crickets chirped for a month, though I did check in with the recruiter for a while (I gave up eventually). Finally the recruiter called me with a one-line brush off "we've decided not to move forward at this time…"
Granted, this was a bad experience, and it won't always be like this. But I'm pretty close to saying "never" to these exams.
I thought Gayle's suggestions were good. She goes so far as to suggest a 90% passage rate - that you should not be giving these tests to people who are unlikely to pass then, using them to confirm, not screen.
This only applies if the take-home interview is obligatory.
I agree that obligatory take-home interviews are unfair (I feel the same about trial periods). They work (they're probably more accurate than a standard interview), but they take too much time from the applicant (and will thus scare away many of the best people).
As an option, however, I totally disagree. There's a significant percentage of good programmers who are ill served by standard interviews. Those people want this.
I agree that obligatory take-home interviews are unfair (I feel the same about trial periods). They work (they're probably more accurate than a standard interview), but they take too much time from the applicant (and will thus scare away many of the best people).
As an option, however, I totally disagree. There's a significant percentage of good programmers who are ill served by standard interviews. Those people want this.
You know, while I agree that these problems are mitigated somewhat by making the exam optional, I don't think this alone solves all the problems associated with this approach. I still think that asking candidates to spend, say, 5-7 hours on something is a big deal, even if you give them a choice to take a different path.
I'm not inherently opposed to technical tests, either in person or take home exams. I believe that many exam-based institutions adhere to a code (probably unwritten) that grants certain rights to the examinee. Think about exams in college, where stakes can actually be quite high.
Think about exams at a university. Typically the subject matter and nature of exam questions is available in advance. The grader is highly competent in the field. An associated study path is available for the exam. The exam will be graded and scored, and those scores will be communicated to the student within a set time frame. Feedback will be provided to the student. The exam is part of achieving a lasting credential, such as credit for a course on a transcript.
None of these things exist in technical exams, and in many ways, this increases the stress. Merely making an exam "optional" doesn't erase all these problems. I think this is the core problem with technical exams, they contain all of the stress for the student, but have none of those rights that I believe exist in universities and other exam-based institutions for a good reason.
I'm not inherently opposed to technical tests, either in person or take home exams. I believe that many exam-based institutions adhere to a code (probably unwritten) that grants certain rights to the examinee. Think about exams in college, where stakes can actually be quite high.
Think about exams at a university. Typically the subject matter and nature of exam questions is available in advance. The grader is highly competent in the field. An associated study path is available for the exam. The exam will be graded and scored, and those scores will be communicated to the student within a set time frame. Feedback will be provided to the student. The exam is part of achieving a lasting credential, such as credit for a course on a transcript.
None of these things exist in technical exams, and in many ways, this increases the stress. Merely making an exam "optional" doesn't erase all these problems. I think this is the core problem with technical exams, they contain all of the stress for the student, but have none of those rights that I believe exist in universities and other exam-based institutions for a good reason.
Yes, that is a very good point. Part of being good at hiring is becoming able to work with different kinds of people with different needs and preferences.
I've had a similar experience. A company I was trying to intern at asked me to write a rails app for them before they would think about giving me an interview. I wrote the app (which all-in took maybe 10 hours) and sent the recruiter a link to the heroku instance the app was running on and the repo on github.
I never got a response. I think three months later I might have gotten a one-liner saying they had gone in "a different direction" or some bullshit like that.
It's insulting and ludicrous for companies to treat prospective employees so disrespectfully.
I never got a response. I think three months later I might have gotten a one-liner saying they had gone in "a different direction" or some bullshit like that.
It's insulting and ludicrous for companies to treat prospective employees so disrespectfully.
This x100. I hate these interviews as one of the first exercises. More than once over the years I have spent 5+ hours doing a coding exercise for a company only to get a one sentence response, telling me that my "code fell short". If I pressed them further they might say my code was "hard to read" or "not what they were looking for". Vague, unhelpful, and pitiful responses. 0 interaction. No chance for a rebuttal.
Its a very lopsided assessment. Normally an in interview you can judge the company while the company judges you. That way in an hour interview you both can probably tell its not a good fit. If I'm applying and coding in python and within 15 minutes of a live coding exercise they tell me I shouldn't use list comprehensions because they are hard to read, I'd be out of there faster than I could measure. If I just spent 5 hours coding to get the same response? I'd get a little upset.
When I give critical feedback I will typically go through line by line and tell the candidate what is wrong and what is good. This is done with the candidate, since nobody is 100% correct so my judgement might be wrong and they can correct me. Many, many companies skip this crucial part and as a candidate you don't know if they are going to do it or not, so why waste your time?
Its a very lopsided assessment. Normally an in interview you can judge the company while the company judges you. That way in an hour interview you both can probably tell its not a good fit. If I'm applying and coding in python and within 15 minutes of a live coding exercise they tell me I shouldn't use list comprehensions because they are hard to read, I'd be out of there faster than I could measure. If I just spent 5 hours coding to get the same response? I'd get a little upset.
When I give critical feedback I will typically go through line by line and tell the candidate what is wrong and what is good. This is done with the candidate, since nobody is 100% correct so my judgement might be wrong and they can correct me. Many, many companies skip this crucial part and as a candidate you don't know if they are going to do it or not, so why waste your time?
Unfortunately, a big part of this problem is that companies face serious liability risks if they give useful feedback.
My experience with tech interviews is that they are actually exams, taken under stressful conditions, with none of the courtesies normally extended to a student.
For instance, in college, or grad school, there is a process for taking an exam. There is typically an affiliated study path, you receive feedback on your performance by a set deadline, and at a good university, someone highly competent grades your exam.
In spite of this, people often describe exams like the bar or their medical boards as the most stressful academic experience they've ever had. As programmers, we have to go up to the whiteboard regularly to take a test, or complete a take-home exam and send it off to who knows who, but we often don't know the subject that will be tested, the competence or credentials of our examiners, whether it will reeve a fair assessment, or even any assessment at all (do they just throw the thing in the trash and say "we decided to go in a new direction"? Truly I have no idea.
That's a huge problem, and it is actually outright harming our industry.
Check this article out
http://www.fastcompany.com/3043082/most-creative-people/why-...
This particular article is about hiring women, but I'm absolutely certain that plenty of men are also deterred from pursuing new tech jobs because they can't stomach the idea of another round of technical testing (with none of the factors I listed above that makes it more fair for the examinee), whether that is white boarding data structure or doing take home projects. I think a lot of people may look at this and decide to just enter a different industry altogether, and I really can't blame them.
I'm on a tangent here, but I really think that tech needs to heal itself, and we're a long way form it.
My experience with tech interviews is that they are actually exams, taken under stressful conditions, with none of the courtesies normally extended to a student.
For instance, in college, or grad school, there is a process for taking an exam. There is typically an affiliated study path, you receive feedback on your performance by a set deadline, and at a good university, someone highly competent grades your exam.
In spite of this, people often describe exams like the bar or their medical boards as the most stressful academic experience they've ever had. As programmers, we have to go up to the whiteboard regularly to take a test, or complete a take-home exam and send it off to who knows who, but we often don't know the subject that will be tested, the competence or credentials of our examiners, whether it will reeve a fair assessment, or even any assessment at all (do they just throw the thing in the trash and say "we decided to go in a new direction"? Truly I have no idea.
That's a huge problem, and it is actually outright harming our industry.
Check this article out
http://www.fastcompany.com/3043082/most-creative-people/why-...
This particular article is about hiring women, but I'm absolutely certain that plenty of men are also deterred from pursuing new tech jobs because they can't stomach the idea of another round of technical testing (with none of the factors I listed above that makes it more fair for the examinee), whether that is white boarding data structure or doing take home projects. I think a lot of people may look at this and decide to just enter a different industry altogether, and I really can't blame them.
I'm on a tangent here, but I really think that tech needs to heal itself, and we're a long way form it.
That is exactly right. Its terrifying. If if you were the first engineer at 2 successful companies and have a strong github it doesn't always seem to factor in.
I would cram for hours before my blackboard interview making sure I knew all sorts of algorithms, just in case. My friend and I would quiz each other. I don't see how memorizing tree transformations relate to a Django application.
It made it so much more difficult to move jobs.
I would cram for hours before my blackboard interview making sure I knew all sorts of algorithms, just in case. My friend and I would quiz each other. I don't see how memorizing tree transformations relate to a Django application.
It made it so much more difficult to move jobs.
I like that they're making the project an optional track, but as a fairly extroverted interviewee I would much prefer talking through code with an interviewer. I feel like the 8 hours of programming I'd do on a take-home project could be assessed with a one-hour conversation walking through code I've already written. (It's especially frustrating when the projects are low-level, like creating a basic CRUD app that uses a lot of libraries, and I've implemented the solution so many times that most of the work on the project is just typing it out. I could just walk the interviewer through one of my many CRUD app side projects.)
Furthermore, collaborating in-person with another engineer tells me more about their engineering culture, and tells them more about whether I'm someone who can collaborate well with others. An hour of pair programming on one engineer's current real-job task would tell us both a lot about each other in a fraction of the time.
Furthermore, collaborating in-person with another engineer tells me more about their engineering culture, and tells them more about whether I'm someone who can collaborate well with others. An hour of pair programming on one engineer's current real-job task would tell us both a lot about each other in a fraction of the time.
A test is only as good as the engineer who glance's at it with the fate of the candidate in his/her hands.
In my experience 90% tests get 10% of the attention of other assessments.
In my experience 90% tests get 10% of the attention of other assessments.
I've been giving applicants home assignments for years now, in 3 companies. I do it as a third step, only after a phone screening and at least one personal interview (that might include some very very light whiteboard coding, but will touch a lot of technical stuff), and only if I find the person to be right for the job.
So out of dozens of CVs, a handful will be invited to a personal interview, and maybe one or two people will actually get the home test. I get that it's time consuming so I try not to waste people's time if I'm not serious about them. On a couple of occasions when people said they're time limited, interviewing for a lot of places, etc - I've allowed them a one hour on site task that is of course simpler.
And it's very rare that people are rejected based solely on the code in the home test, it usually has more factors than that. But if you have two good candidates, it might help tip the balance in favor of one of them.
BTW I don't only review the code. I find that the quality of the documentation (not only comments - I ask for a short text describing the design choices, code structure, etc) is usually one of the best signals. Bad grammar and poor language, complicated descriptions of simple things, focusing on unimportant parts - are huge telltales of problems in a person's thinking. Clear, well versed, concise text usually indicates a smart and pragmatic person.
So out of dozens of CVs, a handful will be invited to a personal interview, and maybe one or two people will actually get the home test. I get that it's time consuming so I try not to waste people's time if I'm not serious about them. On a couple of occasions when people said they're time limited, interviewing for a lot of places, etc - I've allowed them a one hour on site task that is of course simpler.
And it's very rare that people are rejected based solely on the code in the home test, it usually has more factors than that. But if you have two good candidates, it might help tip the balance in favor of one of them.
BTW I don't only review the code. I find that the quality of the documentation (not only comments - I ask for a short text describing the design choices, code structure, etc) is usually one of the best signals. Bad grammar and poor language, complicated descriptions of simple things, focusing on unimportant parts - are huge telltales of problems in a person's thinking. Clear, well versed, concise text usually indicates a smart and pragmatic person.
"At the core of the problem is that this approach can be used to burn a lot of a candidate's time without an equivalent investment from the company."
Maybe a tweak would be to only offer take home projects to candidates only after committing to bring them in for an on-site interview, then the interview consists mostly of reviewing the code from the take home.
(Personally, I think I'd still prefer the "coding in person" option.)
Maybe a tweak would be to only offer take home projects to candidates only after committing to bring them in for an on-site interview, then the interview consists mostly of reviewing the code from the take home.
(Personally, I think I'd still prefer the "coding in person" option.)
After three tries with this process, I've decided that "never" is too soon:
1x Take-home test didn't even get read (this was the nail in the coffin) 1x The interview failed because the salary wasn't competitive (should have asked beforehand about the salary, but didn't have the option; lesson learned) 1x Got an in person interview which led to an offer which was later rescinded (the trip was nice, so I'm not complaining)
Employers in this industry are just not responsible enough to even read through hours worth of work by prospective employees, so while a good idea on paper, the take-home test is a horrible idea in practice. Never again, indeed.
1x Take-home test didn't even get read (this was the nail in the coffin) 1x The interview failed because the salary wasn't competitive (should have asked beforehand about the salary, but didn't have the option; lesson learned) 1x Got an in person interview which led to an offer which was later rescinded (the trip was nice, so I'm not complaining)
Employers in this industry are just not responsible enough to even read through hours worth of work by prospective employees, so while a good idea on paper, the take-home test is a horrible idea in practice. Never again, indeed.
That turns into an advantage for Triplebyte, because you are actually applying to potentially several hundred companies at the same time.
[deleted]
Take home interviews are a great indicator of a company's hubris. "Its so awesome to work here people are going to jump at the chance to do my 3 hour homework assignment".
The problem with them is fundamental: "You will only get someone desperate enough to take your exam."
If the person is qualified they will be swimming in opportunities and will likely throw your exam directly in the trash heap. If they aren't you probably don't want them working for you. I suppose these might work if the entire industry colluded on it but then ... prisoner's dilemma.
Maybe a more useful indicator is weeding out the candidates that didn't say "no thanks." Cause really why are they so desperate for a job and why do they have all this free time? but then ... ethics.
The problem with them is fundamental: "You will only get someone desperate enough to take your exam."
If the person is qualified they will be swimming in opportunities and will likely throw your exam directly in the trash heap. If they aren't you probably don't want them working for you. I suppose these might work if the entire industry colluded on it but then ... prisoner's dilemma.
Maybe a more useful indicator is weeding out the candidates that didn't say "no thanks." Cause really why are they so desperate for a job and why do they have all this free time? but then ... ethics.
> The problem with them is fundamental: "You will only get someone desperate enough to take your exam."
Lots of people do poorly in whiteboard situations. As an employer, you may assert that you don't want any of them, fine. But I don't see the problem in giving people the option of using an alternative testing process.
Lots of people do poorly in whiteboard situations. As an employer, you may assert that you don't want any of them, fine. But I don't see the problem in giving people the option of using an alternative testing process.
Wait - where does it say this is an option? It isn't as far as I can tell - it is how they do it. You either invest 3-9 hours on a "project" before the interview or you don't get the interview.
This isn't "an alternative testing process" as such.
This isn't "an alternative testing process" as such.
The post pretty clearly says:
> To solve the problem of interview anxiety, we're adding a second track to our interview process at Triplebyte. Applicants, if they choose, will be able go through our process by completing programming projects on their own time.
> The project-based track will require a larger time commitment (and we expect lots of people to stick with the standard track for this reason).
> To solve the problem of interview anxiety, we're adding a second track to our interview process at Triplebyte. Applicants, if they choose, will be able go through our process by completing programming projects on their own time.
> The project-based track will require a larger time commitment (and we expect lots of people to stick with the standard track for this reason).
I logged into my TripleByte account and selected it as an option. The UI is still broken, but yes: it's an option. It's a separate track you can choose to be on.
If its an option and they don't exhibit any bias towards candidates that prefer a traditional interview then thats fine, but they should make it clear in the article.
Founder here. We definitely don't and the outcome of both tracks will be treated exactly the same way. As Ammon wrote it: "Anyone who passes our take-home project assessment will get exactly the same service from us as people who do the regular interviews. We'll work hard to find several YC startups they'd be a great fit for, fast track them through the hiring processes, and handle all logistics of flights/accommodations/scheduling."
zing! you got me. I only skimmed the first paragraph. Nah sounds cool. I wish you the best of luck. Let us know if it works.
Ya I totally agree. Especially if a candidate already has a job, and is just casually looking, there is no way they will spend the time.
Also its more than a little disingenuous on the part of the company. It communicates "my time is more valuable than yours . Go do this thing async so if you don't work out I don't have to waste resources on you." It also drives a perverse incentive on the part of the interviewing organization. The only resources you've spent is the time to send them the test if you are unmotivated or are having second thoughts you don't even have to look at the candidates work. But the candidate might spend hours on this thing.
I prefer an interview system where the company and the candidate both care. Commitment is a requirement for everyone.
I prefer an interview system where the company and the candidate both care. Commitment is a requirement for everyone.
[deleted]
I've done this, and liked the process.
Interview with 2 people for an hour or so. Then you get a Rails project, fix 3 bugs, choose and implement 2 features from a list, then 2 features they choose. Come back in a few days and discuss why/how you did what you did.
Interview with 2 people for an hour or so. Then you get a Rails project, fix 3 bugs, choose and implement 2 features from a list, then 2 features they choose. Come back in a few days and discuss why/how you did what you did.
The problem with random or unaligned assignments is you end up with a randomly skilled programmer. Often not bad, but not on point.
The problem with Rosalind or Project Euler assignments is you end up with an excellent theoretical math or bioinformatics programmer. Often not bad, but not on point.
Fundamentally anyone with a degree or experience or a non-trivial github can write code, but you want to test their judgement, their thought process, their comprehension, their style, their knowledge about your business domain. Other than total open field blank slate projects (very little of my time over the last 35 yrs has been spent in that mode) you usually have existing systems and code. So give them a "special" sample of your own code. Shouldn't be too hard to find unless you're literally hiring the first technical employee. Then give it to them a couple days before an interview and inform then you're gonna review that code together, they'll present you with a rewritten, redesigned version, and then review the rewrite together.
If you'll feel bad about making them do "real" work, the best code to send out is some that has been heavily customized to only work some of the time, not properly error check, and intentionally somewhat obfuscated, so I sincerely hope that code thats screwed up to that level has to be intentionally manually generated for the interview. So strip out most of the failure/error detection code, screw up some of the code, maybe intentionally cut and paste an almost identical function in place to see if they clean that up. Some folks like intentional outright errors, like typos, is this the kind of programmer who can't write English? Also wipe most of the comments, put some intentional logic errors in the comments. This can be fun...
If you're looking for non-intro level programmers "everyone knows everyone" and my latest job we didn't talk programming because I was vouched for as knowing what I'm doing from years of coworker experience at a past employer. I'd be moderately offended if someone I worked with for five years asked me to fizzbuzz, either in person or as a take home test.
The problem with Rosalind or Project Euler assignments is you end up with an excellent theoretical math or bioinformatics programmer. Often not bad, but not on point.
Fundamentally anyone with a degree or experience or a non-trivial github can write code, but you want to test their judgement, their thought process, their comprehension, their style, their knowledge about your business domain. Other than total open field blank slate projects (very little of my time over the last 35 yrs has been spent in that mode) you usually have existing systems and code. So give them a "special" sample of your own code. Shouldn't be too hard to find unless you're literally hiring the first technical employee. Then give it to them a couple days before an interview and inform then you're gonna review that code together, they'll present you with a rewritten, redesigned version, and then review the rewrite together.
If you'll feel bad about making them do "real" work, the best code to send out is some that has been heavily customized to only work some of the time, not properly error check, and intentionally somewhat obfuscated, so I sincerely hope that code thats screwed up to that level has to be intentionally manually generated for the interview. So strip out most of the failure/error detection code, screw up some of the code, maybe intentionally cut and paste an almost identical function in place to see if they clean that up. Some folks like intentional outright errors, like typos, is this the kind of programmer who can't write English? Also wipe most of the comments, put some intentional logic errors in the comments. This can be fun...
If you're looking for non-intro level programmers "everyone knows everyone" and my latest job we didn't talk programming because I was vouched for as knowing what I'm doing from years of coworker experience at a past employer. I'd be moderately offended if someone I worked with for five years asked me to fizzbuzz, either in person or as a take home test.
I did one of these when I applied at my most recent position. It took most of the weekend and actually it was a lot of fun. But I doubt I'd do it again if it required more than a few hours of my time. Any test that time consuming is arguably biased against those with families.
Some advice to those doing take home tests: write (good) comments and include tests.
Some advice to those doing take home tests: write (good) comments and include tests.
Wondering if under this model an unethical person could set up a fake interview process with real assignments to get unpaid work done by others who want the job.
As someone who is still technically waiting - after a month - for his phone interview, I like the idea of doing some actual coding. If you have to make people jump through hoops, at least you might try and make the hoops at least somewhat meaningful.
Edit: I just pressed the button on my TripleByte dashboard to switch to the "project-based" track, but then I get redirected straight back to the same "we're sorry" page I've been seeing all along.
Edit: I just pressed the button on my TripleByte dashboard to switch to the "project-based" track, but then I get redirected straight back to the same "we're sorry" page I've been seeing all along.
> The project-based track will require a larger time commitment
This is the only part I take issue with. I tried a take home interview once that took the better part of a weekend and decided I'd never do it again. There just isn't enough time. Basically, if it takes that long, there needs to be a very high chance that I'm getting hired at the end, or it's not worth it.
This is the only part I take issue with. I tried a take home interview once that took the better part of a weekend and decided I'd never do it again. There just isn't enough time. Basically, if it takes that long, there needs to be a very high chance that I'm getting hired at the end, or it's not worth it.
But do you want someone who cannot handle an interview?
Perhaps take-home testing works if you want a "code monkey" who can sit in a box churning out code according to specifications, but I cannot think of a company that doesn't want more. The ability to interact with others in sometimes stressful environments is, imho, an essential skill for all.
There is also the danger that companies will turn these "tests" into work product. I did an interview last year (legal) that requested "writing samples" on very particular and timely subjects. Once I realized the game I asked them to sign a copyright agreement. They got screaming mad and tried to get me to sign all sorts of NDA junk.
Perhaps take-home testing works if you want a "code monkey" who can sit in a box churning out code according to specifications, but I cannot think of a company that doesn't want more. The ability to interact with others in sometimes stressful environments is, imho, an essential skill for all.
There is also the danger that companies will turn these "tests" into work product. I did an interview last year (legal) that requested "writing samples" on very particular and timely subjects. Once I realized the game I asked them to sign a copyright agreement. They got screaming mad and tried to get me to sign all sorts of NDA junk.
It seems as part of the take home that you spend 45 minutes discussing your implementation and defending your choices in person with the interviewer. This is a much more realistic "on the job" stressor than being able to code out loud / on a white board.
I view these as a good litmus test for the company too. If they won't answer a question about the problem, or think it's very self explanatory, it might mean they're not the best at communicating.
I recently did a clojure problem where in the gist itself I asked for feedback and showed my thought process, but crickets until a month later when I followed up and they said they had moved on. I think it was for the best.
A good preventative measure might be agreeing beforehand on a time to discuss the solution. I know I certainly will next time.
Also hiring managers need to ACTUALLY read code. There have even been times I've asked if they've read my code before doing a lengthy drive, they've said yes, only to realize upon getting there they haven't. If you won't take some time to assess the code I've written and put my name on, I don't think I should be expected to drive 3 hours to meet you.
I recently did a clojure problem where in the gist itself I asked for feedback and showed my thought process, but crickets until a month later when I followed up and they said they had moved on. I think it was for the best.
A good preventative measure might be agreeing beforehand on a time to discuss the solution. I know I certainly will next time.
Also hiring managers need to ACTUALLY read code. There have even been times I've asked if they've read my code before doing a lengthy drive, they've said yes, only to realize upon getting there they haven't. If you won't take some time to assess the code I've written and put my name on, I don't think I should be expected to drive 3 hours to meet you.
This is presented as a weeding out technique but it is actually a negotiation step. This allows the employer to establish precedent that you work off hours from home. This allows the employer to identify candidates that are willing to do whatever it takes for the job. These are the same people that won't do hardball negotiations for salary, employment terms or working conditions. Rather than weeding out the people who will fail the test, it weeds out the people who understand that asking for homework is unreasonable.
My last two phone interviews ended with the recruiter / hiring manager explaining that the next step was a take home assignement or test. I said OK, but haven't done either one. Next time I will politely tell them that they can look at code samples in my gthub but I am not spending my time jumping through their hoops just for the chance to have an in person interview.
It's bad enough that we have to burn an entire day to do an in person interview...there shouldn't also be homework.
My last two phone interviews ended with the recruiter / hiring manager explaining that the next step was a take home assignement or test. I said OK, but haven't done either one. Next time I will politely tell them that they can look at code samples in my gthub but I am not spending my time jumping through their hoops just for the chance to have an in person interview.
It's bad enough that we have to burn an entire day to do an in person interview...there shouldn't also be homework.
My strategy was to tell them something like "First I'm going to interview at all the companies that don't require take home assignments, and then I'll get back to you if I don't have a job by then".
This. Happened with my most recent round of interviews. I took an initial stab at one assignment, moved onto other opportunities, and landed an offer before getting back to the project work.
At least the take-home work I did perform in my second-to-last job search provided good, compact example code for my last search.
At least the take-home work I did perform in my second-to-last job search provided good, compact example code for my last search.
Incredible to read so many cases here where people have done a take-home exercise and heard nothing back. I had assumed that the normal practice would be to give a take-home exercise as part of prep for an in person interview, so if you do the work you're at least guaranteed the opportunity to talk about it.
Take-home interviews are a perfect match for startups - they really reflect actual working conditions. One way or another, you'll be taking home your work every day! :)
I see people always saying this online, but the vast majority of engineers I talk to in-person work very sane hours (9-5, 10-6) and never take their work home with them.
The problem I find with these is that while solving a problem might take 3 hours, you can spend an essentially unlimited amount of time polishing code - writing comments/javadocs, neatening up the code, writing more unit tests etc. etc.
Is there data on the efficacy of non traditional coding interviews vs traditional ones?
I'm all for improving the process, but as someone who at times has struggled on a coding interview, I believe that the onus to improve should be on me the candidate.
I'm all for improving the process, but as someone who at times has struggled on a coding interview, I believe that the onus to improve should be on me the candidate.
I hate take-home interviews and rarely finish them. Especially when a candidate has many options or places a high value on actually getting to meet possible future team members (as I do), a two or three-hour assignment take-home assignment is a lot to ask. I would rather get to show previous projects and explain context and what I might do differently now.
So you do the test, but you're not guaranteed a position even if your code is amazing.
Hell, in my experience you're not even guaranteed anyone will read it.
This is just a bad idea in an industry full of irresponsible employers. I'm not saying that Triplebyte is in any way irresponsible, but the rest of the industry has already ruined this avenue.
Hell, in my experience you're not even guaranteed anyone will read it.
This is just a bad idea in an industry full of irresponsible employers. I'm not saying that Triplebyte is in any way irresponsible, but the rest of the industry has already ruined this avenue.
Triplebyte co-founder here. We understand how frustrating it'd be to spend time on a project and then have no one read it, we won't be doing that. In our process, you schedule a time to talk with us about the code you've written. We'll go through it together and provide feedback.
We're not replacing talking to another person part of the interview, we're just trying to remove the awkwardness of having someone watch you code (for those programmers who want it).
We're not replacing talking to another person part of the interview, we're just trying to remove the awkwardness of having someone watch you code (for those programmers who want it).
This seems like a wonderful idea.
I hate coding interviews, because I freeze up and I look like a total idiot and cannot do even the simplest things. Same thing at exams in school or university - my mind just went blank and into a noisy self-rumination loop. I guess it's called anxiety.
I interviewed with Google last week and bombed it, even though the problem was quite simple and I would have solved it wonderfully without all that anxiety.
My mind just can't produce any sensible thoughts when someone's breathing over my neck and there's a timer. When I'm solving the problem in real time and talking about it, I have to stick with the split decisions that I verbalize and build on top of them, even though further down the road I realise they're not optimal and this contributes to the anxiety even more.
It's hard to 'refactor' your ideas during a 45m coding interview, but this is exactly the process we go through when we write 'real' code - we try a thing, then we improve it by refactoring, we optimize it, we find and fix corner cases, etc.
A take-home problem, on the other hand, would be totally cool. I have time to think about the problem, come up with solutions, optimize, unit test - do the real programming thing which I'm being hired for.
I would then gladly discuss and explain the code with the interviewer.
If then I would be given the task to extend the program with a new feature, then I would be familiar with the data structures and algorithms used and would probably find it much easier to extend, than starting with a blank file and figuring it out on the spot.
But I guess not everyone agrees with the home interviews - some think it's a waste of time and I'd have to agree..
So I guess the optimal solution is to offer the option of on-site coding interview or a take-home problem.
People like me would take the problem home, build it and shine at it, others, who's minds are sharpened by the adrenaline would take the on-site 1hr coding challenge.
I hate coding interviews, because I freeze up and I look like a total idiot and cannot do even the simplest things. Same thing at exams in school or university - my mind just went blank and into a noisy self-rumination loop. I guess it's called anxiety.
I interviewed with Google last week and bombed it, even though the problem was quite simple and I would have solved it wonderfully without all that anxiety.
My mind just can't produce any sensible thoughts when someone's breathing over my neck and there's a timer. When I'm solving the problem in real time and talking about it, I have to stick with the split decisions that I verbalize and build on top of them, even though further down the road I realise they're not optimal and this contributes to the anxiety even more.
It's hard to 'refactor' your ideas during a 45m coding interview, but this is exactly the process we go through when we write 'real' code - we try a thing, then we improve it by refactoring, we optimize it, we find and fix corner cases, etc.
A take-home problem, on the other hand, would be totally cool. I have time to think about the problem, come up with solutions, optimize, unit test - do the real programming thing which I'm being hired for.
I would then gladly discuss and explain the code with the interviewer.
If then I would be given the task to extend the program with a new feature, then I would be familiar with the data structures and algorithms used and would probably find it much easier to extend, than starting with a blank file and figuring it out on the spot.
But I guess not everyone agrees with the home interviews - some think it's a waste of time and I'd have to agree..
So I guess the optimal solution is to offer the option of on-site coding interview or a take-home problem.
People like me would take the problem home, build it and shine at it, others, who's minds are sharpened by the adrenaline would take the on-site 1hr coding challenge.
As others have mentioned, the trade-offs can be pretty complicated. Perhaps the take-home is always better for you, I won't try to argue against your personal preferences, I'm simply going to list a few reasons why it turns out worse for a lot of people.
For one thing, as expected, the take-home questions are typically far larger than in-person interview questions. When you go into the office you're generally asked, in my experience, much simpler questions; you might get a simple FizzBuzz-like starter, and a few slightly more difficult ones. All in, it takes a few hours (Google and the like can be an exception to that, but I'll get to that after - I'm comparing smaller companies right now).
The take-home work I've seen has generally been suggested by the company to take 8 hours, but typically been a bit more; they've been along the lines of build a fairly simple crud (I've had one that was a calendar, another that's a stream, etc.) with a bootstrap-or-similar UI, back-end validations, and unit tests for the initial commit, implement these 3-5 features on top of it in separate commits (tested, also standard bootstrap-or-similar UI, proper validation, etc.) And then, simply provide access to the repo.
At first, this didn't seem wrong, in fact I figured "well, I guess it's a good way for them to see that you understand everything". But it came with several annoyances. For one thing, since each feature is a commit, unless you specifically try to prevent it they can see things like how long each part took; even if not consciously, this makes people try to get the work done quickly, without taking many breaks. It sort of reintroduces the "hard to 'refactor' your ideas during a 45m coding interview" problem. Another annoyance is that sometimes (and judging by other comments here, even often) they don't respond at all. The silence is unpleasant, to say the least. In-person, you can at least try to read tone and facial expressions.
The other problem is one company I applied with threw a 10+ hour take-home without any warning before the interview. So a bit over an hour in their office, and another 10+ later. This is difficult if you're working another job. It's very difficult if you have a couple such take-homes at the same time, and a full-time, in office job. Scheduling a few interviews around work, and other commitments, is pretty feasible, but throwing in dozens of hours of additional coding is more difficult.
With companies like Google, you know ahead of time approximately how long it will take. You can take time off accordingly. The process is known going in, and that's great.
Basically, I think that for many people, myself included, white-board interviews are very stressful and frightening, but not quite as much so as: scheduling lots of extra time around your life, budgeting X extra hours because they take longer than suggested, building out the (much larger) projects, still having the time-constraint/expectancy issues, and after all of that not always even getting a message/comment afterward.
Both interview processes suck.
For one thing, as expected, the take-home questions are typically far larger than in-person interview questions. When you go into the office you're generally asked, in my experience, much simpler questions; you might get a simple FizzBuzz-like starter, and a few slightly more difficult ones. All in, it takes a few hours (Google and the like can be an exception to that, but I'll get to that after - I'm comparing smaller companies right now).
The take-home work I've seen has generally been suggested by the company to take 8 hours, but typically been a bit more; they've been along the lines of build a fairly simple crud (I've had one that was a calendar, another that's a stream, etc.) with a bootstrap-or-similar UI, back-end validations, and unit tests for the initial commit, implement these 3-5 features on top of it in separate commits (tested, also standard bootstrap-or-similar UI, proper validation, etc.) And then, simply provide access to the repo.
At first, this didn't seem wrong, in fact I figured "well, I guess it's a good way for them to see that you understand everything". But it came with several annoyances. For one thing, since each feature is a commit, unless you specifically try to prevent it they can see things like how long each part took; even if not consciously, this makes people try to get the work done quickly, without taking many breaks. It sort of reintroduces the "hard to 'refactor' your ideas during a 45m coding interview" problem. Another annoyance is that sometimes (and judging by other comments here, even often) they don't respond at all. The silence is unpleasant, to say the least. In-person, you can at least try to read tone and facial expressions.
The other problem is one company I applied with threw a 10+ hour take-home without any warning before the interview. So a bit over an hour in their office, and another 10+ later. This is difficult if you're working another job. It's very difficult if you have a couple such take-homes at the same time, and a full-time, in office job. Scheduling a few interviews around work, and other commitments, is pretty feasible, but throwing in dozens of hours of additional coding is more difficult.
With companies like Google, you know ahead of time approximately how long it will take. You can take time off accordingly. The process is known going in, and that's great.
Basically, I think that for many people, myself included, white-board interviews are very stressful and frightening, but not quite as much so as: scheduling lots of extra time around your life, budgeting X extra hours because they take longer than suggested, building out the (much larger) projects, still having the time-constraint/expectancy issues, and after all of that not always even getting a message/comment afterward.
Both interview processes suck.
There's a lot of truth in what you write and I definetely see the problem.
One solution is to have the take home interview as the final step in the process - after having screened the candidate and discussed general technology/framework/concepts without coding.
When I did interviews, I found that just by discussing things like thread synchronization and let them explain the projects that they've worked on would weed out most of the candidates which weren't a fit with the position. Then the take-home problem would make a lot more sense for both the employer and employee.
Or another method: You take home a problem, implement it, then present the solution to multiple potential employers. The problem would have to be unique (so that there are no ready made solutions online) and trusted by the participating employers.
And I think this is exactly what OP is doing, and that's why it's great.
One solution is to have the take home interview as the final step in the process - after having screened the candidate and discussed general technology/framework/concepts without coding.
When I did interviews, I found that just by discussing things like thread synchronization and let them explain the projects that they've worked on would weed out most of the candidates which weren't a fit with the position. Then the take-home problem would make a lot more sense for both the employer and employee.
Or another method: You take home a problem, implement it, then present the solution to multiple potential employers. The problem would have to be unique (so that there are no ready made solutions online) and trusted by the participating employers.
And I think this is exactly what OP is doing, and that's why it's great.
I agree with all of those sentiments.
At the same time, I prefer the idea of presenting previous open source work instead of having a group of potential employers agree on a project that they can all judge. One benefit is that the work is less arbitrary, potentially useful for others. Another is it's more likely to be unique than an assignment is. Finally, do I want to work for any of those other employers?
Applying to a group that way would seem odd to me; part of me thinks that choosing companies is important, but another thinks that there's some chance that that's just because I haven't tried it.
Which, of course, leads me to agree that the OP is great as an experiment - it might well be a better solution in many cases. The fact that, in the OP, it's opt-in is a big deal as well. Though I'm still very skeptical of take-home interviews in general.
At the same time, I prefer the idea of presenting previous open source work instead of having a group of potential employers agree on a project that they can all judge. One benefit is that the work is less arbitrary, potentially useful for others. Another is it's more likely to be unique than an assignment is. Finally, do I want to work for any of those other employers?
Applying to a group that way would seem odd to me; part of me thinks that choosing companies is important, but another thinks that there's some chance that that's just because I haven't tried it.
Which, of course, leads me to agree that the OP is great as an experiment - it might well be a better solution in many cases. The fact that, in the OP, it's opt-in is a big deal as well. Though I'm still very skeptical of take-home interviews in general.
Why is it ok with everyone that triplebyte is using desperate job seekers as lab rats?
"Hey! Stop measuring me and give me job! I'm smart you asshole."
~ every software engineer in the world
Anyone else realize we're the only ones who complain about this shit?
~ every software engineer in the world
Anyone else realize we're the only ones who complain about this shit?
The interview process if kind-of crazy. I've been looking for a job for the last few weeks and been on a few dozen interviews. 90% of my experience is this:
1) They have a very-long interview process. Start to end is months, with weeks between communications. 2) Each phase, when there are often from 6-10, can take anytime from 2-8 hours. 3) Projects are always either right off the bat, and lengthy, or a surprise at the end of the interview.
1) They have a very-long interview process. Start to end is months, with weeks between communications. 2) Each phase, when there are often from 6-10, can take anytime from 2-8 hours. 3) Projects are always either right off the bat, and lengthy, or a surprise at the end of the interview.
I considered a job out in SF working for a decently well-known guy, Andrew Chen. I had gone through 4 coding interviews progressively working my way up their ladder to their CTO. Now, each of these I did well on but they didn't like how fast I went. Despite them saying "Our fastest done was 1hr!", they didn't seem to appreciate me going fast and came to the conclusion that I probably didn't know how to write production code -- just a quick hacker. Actually, it was rather annoying because their "top" iOS dev didn't even understand basic shit I was doing. He'd have to keep stopping me and being like wait -- why'd you do that? Yet as I worked my way up, I really enjoyed a couple of their engineers and CTO. I didn't have to hold THEIR hand during a code interview.
So I was an iOS developer getting considered for Android position, and they said "hey it's great you can code fast & all, but can you learn fast?" and told me to create an Android app that searched images on Google. I sighed because I had already been through 4 coding interviews, but oh well, this will be it I thought. Since I wanted to do a good job, I spent two hard days working on it basically all day every day. Since I was new to Android world, I had to learn that while being productive. You can see the result of my efforts here: https://github.com/joslinm/android-image-search-example. I thought I did a nice job because rather than using a HTTP library, I read the streams myself to give nice progress indicators for each image (which persisted even thru phone rotates).
I give it to them and they say, "ok seems to be working." After that, I was invited to SF to interview with them in person -- actually, interview is the wrong word, I would be working with them for 2 full days. So we did that. When I wasn't coding, I was being grilled about my work history (perfectly acceptable, but why didn't we do this earlier?). Finally, after that, I was told they had come to their decision. Thinking I was a shoe-in, I went to go meet with Andrew. Andrew's decision: "You're too entrepreneurial."
Now let's examine this for a quick second. They had come to a pretty fair conclusion that I was too entrepreneurial after talking to me. They realized, hey this kid is a go-getter and probably isn't the greatest fit for our #7. Yet this was AFTER copious amounts of coding and wasting my time doing all these code interviews and even joining them to code in-house. If they had just went about a pretty normal interview, they would have discovered this fact way earlier and not wasted either of our time. So why waste my time forcing me to code before they do their diligence? Because he's Andrew Chen; you can google his name and discover who he is. It would be a "honor" to work for such a prestigious start-up pundit. In other words, he can get away with it.
For those of you considering a start-up and actually have skill.. don't get taken advantage of by these bogus work projects. The employer wins because he's not paying you and gets a lot of evaluation for free. And if they do insist on you working with them for a couple days, you insist on getting paid for those couple days.
So I was an iOS developer getting considered for Android position, and they said "hey it's great you can code fast & all, but can you learn fast?" and told me to create an Android app that searched images on Google. I sighed because I had already been through 4 coding interviews, but oh well, this will be it I thought. Since I wanted to do a good job, I spent two hard days working on it basically all day every day. Since I was new to Android world, I had to learn that while being productive. You can see the result of my efforts here: https://github.com/joslinm/android-image-search-example. I thought I did a nice job because rather than using a HTTP library, I read the streams myself to give nice progress indicators for each image (which persisted even thru phone rotates).
I give it to them and they say, "ok seems to be working." After that, I was invited to SF to interview with them in person -- actually, interview is the wrong word, I would be working with them for 2 full days. So we did that. When I wasn't coding, I was being grilled about my work history (perfectly acceptable, but why didn't we do this earlier?). Finally, after that, I was told they had come to their decision. Thinking I was a shoe-in, I went to go meet with Andrew. Andrew's decision: "You're too entrepreneurial."
Now let's examine this for a quick second. They had come to a pretty fair conclusion that I was too entrepreneurial after talking to me. They realized, hey this kid is a go-getter and probably isn't the greatest fit for our #7. Yet this was AFTER copious amounts of coding and wasting my time doing all these code interviews and even joining them to code in-house. If they had just went about a pretty normal interview, they would have discovered this fact way earlier and not wasted either of our time. So why waste my time forcing me to code before they do their diligence? Because he's Andrew Chen; you can google his name and discover who he is. It would be a "honor" to work for such a prestigious start-up pundit. In other words, he can get away with it.
For those of you considering a start-up and actually have skill.. don't get taken advantage of by these bogus work projects. The employer wins because he's not paying you and gets a lot of evaluation for free. And if they do insist on you working with them for a couple days, you insist on getting paid for those couple days.
That sucks. There was a time when being entrepreneurial was added to job descriptions as a nice to have. I went through something similar a long time ago; the reasoning was just as specious.
Good luck.
Good luck.
Thanks -- specious is a nice word. Never heard that one before. Looks like I got even more out of that experience ;)
I would never take this option.
At my current workplace, I have the tools I need to work. Most of my softwares licences are under my employer's name. I have access to my snippets, my previous projects and many build processes (minifying my code, preprocessing my CSS, etc.). My workflow is great.
If you give me a project to do at home, I can't use any of that.
I would have to spent hours working either from my bed, my dining table or in a coffee, on a laptop. I would also be required to work on this project after the 40h/week that I already do.
I'll take the regular interview any time.
One of my worst interview process was one time when they required me to do a project of 20+h in a week. They knew I was employed elsewhere, so they let me give it Monday instead of Friday. This was ridiculous.
Otherwise, everything was doing very well. However, the experience really discouraged me from working there.
But I guess it's not all that bad, because I most likely dodged a bullet. If they can't even give correct deadlines to future employees, I can't imagine how they threat their own employees.
At my current workplace, I have the tools I need to work. Most of my softwares licences are under my employer's name. I have access to my snippets, my previous projects and many build processes (minifying my code, preprocessing my CSS, etc.). My workflow is great.
If you give me a project to do at home, I can't use any of that.
I would have to spent hours working either from my bed, my dining table or in a coffee, on a laptop. I would also be required to work on this project after the 40h/week that I already do.
I'll take the regular interview any time.
One of my worst interview process was one time when they required me to do a project of 20+h in a week. They knew I was employed elsewhere, so they let me give it Monday instead of Friday. This was ridiculous.
Otherwise, everything was doing very well. However, the experience really discouraged me from working there.
But I guess it's not all that bad, because I most likely dodged a bullet. If they can't even give correct deadlines to future employees, I can't imagine how they threat their own employees.
I've done quite a few "at home" programming challenges [1][2][3] as well as hacking challenges and most of my experiences have been overwhelmingly negative. The challenge is that "programming" is about 20% writing code and 80% other stuff (requirements, design, testing, security, documentation, meetings, beer, etc...). It seems like the proposed solution is not addressing the problem: Explaining how to solve a programming challenges you may have never heard of on the spot.
I do agree that interviews add a level of pressure, but what you're trying to gauge thought process. How does the candidate think about solving the problem; Can they iterate; Can we lower the space and/or time complexity; Can this be solved recursively, Are they receptive to input? A take home assignment does't help you figure any of that out. Usually, the team who is interviewing already knows the answer to the specific problem and all of the intricacies (based on having to solve it on an internal project or just because it's a question they've asked before), but the goal is to see someone thinking in real-time.
In my career, I've never had a time where I go off on my own, work on something and come back to the team with a solution. Even when I was writing TI-83 programs in High School to solve all of my science homework and finish 50 minute tests in 10 minutes; I still had classmates asking me about what I was doing along the way. There is always discussion and I feel like by sending me off on my own, all of that goes out of the window. You can't verify if I copied the code from anywhere else, you don't know if I had my buddy who is an engineer at another company break down the answer for me to spit it back to you, and you don't know if I paired with someone else to get the assignment complete. I'm obviously biased based on my experiences doing over 40 interviews in Silicon Valley/SF. My goal for this type of interview question would be to understand how the candidate thinks; I don't care if you write your answer in Brainfk[4].
[1] - https://github.com/joeblau/sample-elevator-control-system
[2] - https://github.com/joeblau/sample-top-ten-tweets
[3] - https://github.com/joeblau/sample-url-shortner
[4] - https://en.wikipedia.org/wiki/Brainfuck
I do agree that interviews add a level of pressure, but what you're trying to gauge thought process. How does the candidate think about solving the problem; Can they iterate; Can we lower the space and/or time complexity; Can this be solved recursively, Are they receptive to input? A take home assignment does't help you figure any of that out. Usually, the team who is interviewing already knows the answer to the specific problem and all of the intricacies (based on having to solve it on an internal project or just because it's a question they've asked before), but the goal is to see someone thinking in real-time.
In my career, I've never had a time where I go off on my own, work on something and come back to the team with a solution. Even when I was writing TI-83 programs in High School to solve all of my science homework and finish 50 minute tests in 10 minutes; I still had classmates asking me about what I was doing along the way. There is always discussion and I feel like by sending me off on my own, all of that goes out of the window. You can't verify if I copied the code from anywhere else, you don't know if I had my buddy who is an engineer at another company break down the answer for me to spit it back to you, and you don't know if I paired with someone else to get the assignment complete. I'm obviously biased based on my experiences doing over 40 interviews in Silicon Valley/SF. My goal for this type of interview question would be to understand how the candidate thinks; I don't care if you write your answer in Brainfk[4].
[1] - https://github.com/joeblau/sample-elevator-control-system
[2] - https://github.com/joeblau/sample-top-ten-tweets
[3] - https://github.com/joeblau/sample-url-shortner
[4] - https://en.wikipedia.org/wiki/Brainfuck
> Propranolol is used by musicians, actors, and public speakers for its ability to treat anxiety symptoms activated by the sympathetic nervous system.
- https://en.wikipedia.org/wiki/Propranolol#Society_and_cultur...
Can ask your doctor about this
- https://en.wikipedia.org/wiki/Propranolol#Society_and_cultur...
Can ask your doctor about this
This sounds like a brilliant idea. I hope that you are also including homework assignments that cover (1) communication skills (especially WRITTEN communication) and (2) personality. In my experience, those two areas are far more critical to success than raw coding skills. Also, everyone should pass the "no assholes" test, if there is such a thing.
I hope that you expect the same amount of prerequisite work for all of your candidates (marketing, finance, sales).
I hope that you expect the same amount of prerequisite work for all of your candidates (marketing, finance, sales).
My personal favorite method is similar to this and we use it at my current company.
The big difference of course is that we pay for the candidates time. Not everyone has 5-20 hours to burn on a project in terms of free time. It's a big commitment, especially for the folks with families.
Some key benefits of a project for our interview process:
* We get to see a candidates programming skills with a greenfield project. This gives us a chance to analyze all aspects of development (organizing code, creating tests, using best practices, ect.).
* Candidates get to act as if they are on the team, so they are encouraged to ask questions, get feedback, ect. throughout the project. This gives us a chance to see how well they communicate in addition to their programming skills, which is REALLY important to us, since we're an entirely distributed team.
* The project is not throwaway for the candidate nor us. We usually have an ongoing list of libraries and tools that we need internally for our projects, so the candidate is actually contributing to our mission. This also justifies paying a candidate for their time
The big difference of course is that we pay for the candidates time. Not everyone has 5-20 hours to burn on a project in terms of free time. It's a big commitment, especially for the folks with families.
Some key benefits of a project for our interview process:
* We get to see a candidates programming skills with a greenfield project. This gives us a chance to analyze all aspects of development (organizing code, creating tests, using best practices, ect.).
* Candidates get to act as if they are on the team, so they are encouraged to ask questions, get feedback, ect. throughout the project. This gives us a chance to see how well they communicate in addition to their programming skills, which is REALLY important to us, since we're an entirely distributed team.
* The project is not throwaway for the candidate nor us. We usually have an ongoing list of libraries and tools that we need internally for our projects, so the candidate is actually contributing to our mission. This also justifies paying a candidate for their time
I've wasted too much time on take-home interviews. I'll never do one again. All the contracts I've ever gotten never required one.
My old employer Thoughtworks does this.
http://www.thoughtworks.com/careers/application-process
It's a fantastic way to vet candidates and there are multiple exits to the process. For obviously bad submissions, it's probably a no-hire, but I've even seen incomplete assignments get someone to the next level and then a hire.
Some people get sent to ThoughtWorks University which basically trains them to code, work in an agile environment and how to be a good consultant. Everyone that goes says it is a great experience.
http://www.thoughtworks.com/careers/application-process
It's a fantastic way to vet candidates and there are multiple exits to the process. For obviously bad submissions, it's probably a no-hire, but I've even seen incomplete assignments get someone to the next level and then a hire.
Some people get sent to ThoughtWorks University which basically trains them to code, work in an agile environment and how to be a good consultant. Everyone that goes says it is a great experience.
We (Sauce Labs, Mobile Team) are experimenting with take-home tests as a first-contact screen. I'm familiar with the take home exam that takes a week to solve, so we went in a different direction.
We schedule time with candidates, send it to them at the right moment, and expect them to send in a solution two hours later.
Before we tried it on candidates, we all did our own test, and we confirmed it could be done in less than two hours. (Granted, we are very familiar with the problems we care about, but we usually get complete solutions from candidates.)
So far the results have been fascinating. The test is conceptually super easy, deliberately so, but with a very wide set of possible implementations. It is more about tying together moving parts in the right design. The goal was to do the opposite of the algorithmic brainteasers you normally get in interviews -- it's a miniaturized version of the problems we actually solve all day. Hopefully, it's a good test of what kind of coder the candidate is.
But we're still working on it and refining it.
We schedule time with candidates, send it to them at the right moment, and expect them to send in a solution two hours later.
Before we tried it on candidates, we all did our own test, and we confirmed it could be done in less than two hours. (Granted, we are very familiar with the problems we care about, but we usually get complete solutions from candidates.)
So far the results have been fascinating. The test is conceptually super easy, deliberately so, but with a very wide set of possible implementations. It is more about tying together moving parts in the right design. The goal was to do the opposite of the algorithmic brainteasers you normally get in interviews -- it's a miniaturized version of the problems we actually solve all day. Hopefully, it's a good test of what kind of coder the candidate is.
But we're still working on it and refining it.
Putting any time constraints on a dev test is a bad idea in my opinion. It's not that I can't finish the test in the allotted time, but knowing that I am being timed will make my brain race and do stupid things. Personally, I don't mind take home tests that take more than 2 hours, as long as they reflect the actual work I would be doing in that position.
We consistently do this in Sandglaz whenever we interview for a technical position. It most closely aligns with how work is done in real life, and we give candidates a reasonable amount of time to do the project. We ask them to document what they would do if they had more time. It is more of a way of peering into how they organize code, their understanding of technical debt and prototyping, and their ability to write unit tests.
It's been invaluable in determining candidate fit.
It's been invaluable in determining candidate fit.
Historically, I've tended to do poorly in in-person interviews. I felt my critical thinking and problem solving skills plummet to a fraction of what I am capable of. I initially thought that with enough real-world interviewing experience, I could familiarize myself to the stress, but that never really happened. Interviews tend to be few and far between, so the familiarity never really sticks.
I usually perform better on take home interviews, but 90% the time I'm unwilling to dedicate what are usually days for just a chance to be accepted at a company I may not even want to join. I think many employers use the take home interview as a screener without realizing they need to first cultivate in the candidate the enthusiasm and willingness to complete it.
Some ways companies can encourage me to actually complete the take home interview:
- Provide compelling information about the job (estimated pay/equity, meeting the team, evaluating culture-fit, getting me excited about the problem, etc). Lots of companies save this step for the courting process that happens after the technical screen.
- Pay me a modest amount just to complete it, regardless if I pass or not. I don't care about the money, but at least it won't feel like wasted time.
- Make it way shorter, but then it might be useless.
--
What I'd prefer instead of all this interviewing is still:
1) meet the team, evaluate culture fit, etc.
2) discuss past projects in detail, maybe do a code evaluation if relevant
3) contract for the company for 2-4 weeks at market rate pay. Hell, make a 10-week "internship" out of it, whatever.
4) receive offer (or not)
This is not always practical or scalable, but I've gotten offers 100% of the time this way, including at YC companies and other startups. And from the company's perspective, I think it extricates any unreasonable expectations they have from prospective employees. It's always tempting to look for unicorns when you have hundreds of resumes to choose from, but when you have a likable contractor doing good work, there's no reason not to give him/her an offer. Plus, the employer can evaluate the single most relevant skill in an employee -- the ability to learn.
I usually perform better on take home interviews, but 90% the time I'm unwilling to dedicate what are usually days for just a chance to be accepted at a company I may not even want to join. I think many employers use the take home interview as a screener without realizing they need to first cultivate in the candidate the enthusiasm and willingness to complete it.
Some ways companies can encourage me to actually complete the take home interview:
- Provide compelling information about the job (estimated pay/equity, meeting the team, evaluating culture-fit, getting me excited about the problem, etc). Lots of companies save this step for the courting process that happens after the technical screen.
- Pay me a modest amount just to complete it, regardless if I pass or not. I don't care about the money, but at least it won't feel like wasted time.
- Make it way shorter, but then it might be useless.
--
What I'd prefer instead of all this interviewing is still:
1) meet the team, evaluate culture fit, etc.
2) discuss past projects in detail, maybe do a code evaluation if relevant
3) contract for the company for 2-4 weeks at market rate pay. Hell, make a 10-week "internship" out of it, whatever.
4) receive offer (or not)
This is not always practical or scalable, but I've gotten offers 100% of the time this way, including at YC companies and other startups. And from the company's perspective, I think it extricates any unreasonable expectations they have from prospective employees. It's always tempting to look for unicorns when you have hundreds of resumes to choose from, but when you have a likable contractor doing good work, there's no reason not to give him/her an offer. Plus, the employer can evaluate the single most relevant skill in an employee -- the ability to learn.
One thing I always found interesting in these discussions is how the same people who are quick to cite The Mythical Man-Month when it comes to the futility of adding new people to a late project will also claim that working 1-3 hours on some trivial problem is "doing free work" for a company. How much value do you think your 3 hours of work with absolutely no context can possibly add to the company? A company trying to get meaningful work done by secretly farming it out to interviewees is beyond ludicrous. Interviewers are only interested in finding good hires, not in conning you into doing their homework for them.
I'm not interested in how much value the company gets out of it; it's how much it costs me that matters.
And anecdotally, 1-3 hours is...optimistic.
And anecdotally, 1-3 hours is...optimistic.
In 2008, I was hired to work on a tech startup from Australia, while I was living in Brazil. After I went through an initial quiz and a few interviews, they were willing to give me a shot by giving me (and paying for) a small project to code. Only after succeeding at it, they hired me, sponsored my visa and paid for all the relocation costs. In your case, I really think this approach would be a good bet in cases that the YC company is willing to sponsor the visa from the potential employee, so giving him a paid project before even flying him, might be a good filter.
This is great, but I feel like take-home interviews are better for a first-level filter. I went through this when I applied to Thoughtworks, but it definitely wasn't the only interview. This may have been because I was in a different state also though, so before they would pay to fly me out, they let me pick out of a few problems and then email them in within 72-hours.
To me the value in a take-home test is in smoke testing what the applicant said they are familiar with. We once accidentally hired someone once who talked the talk but didn't even know SSH.
My test involves a basic task (that is related and already implemented in our codebase). You have to check out a skeleton project, add the solution, build it, and deploy it. The test takes me about 30 minutes and most applicants spend 60-90 on it. Everyone on the team reviews the test and the criteria is essentially "do you want to work with this person". It's very rare for someone to pass the test (which occurs between a phone screen and an in-person interview) and not get an offer. The exceptions have all been around compensation issues.
My test involves a basic task (that is related and already implemented in our codebase). You have to check out a skeleton project, add the solution, build it, and deploy it. The test takes me about 30 minutes and most applicants spend 60-90 on it. Everyone on the team reviews the test and the criteria is essentially "do you want to work with this person". It's very rare for someone to pass the test (which occurs between a phone screen and an in-person interview) and not get an offer. The exceptions have all been around compensation issues.
Here's the best approach I've seen yet to the take home interview (and interviewing software engineering candidates in general).
Here's a git repo, a problem statement and a slack channel to ask questions. You can use any tools you like and spend as much time as you like on the project for the next week.
----
Employer's perspective: You get to see some code, see how they approach a problem, see how they use their tools, and get a sense for what it's like to work with them.
Candidates's Perspective: You get to use the tools you're already comfortable with, you can set your own pace, and you get a sense of what it's like to work with the team.
Here's a git repo, a problem statement and a slack channel to ask questions. You can use any tools you like and spend as much time as you like on the project for the next week.
----
Employer's perspective: You get to see some code, see how they approach a problem, see how they use their tools, and get a sense for what it's like to work with them.
Candidates's Perspective: You get to use the tools you're already comfortable with, you can set your own pace, and you get a sense of what it's like to work with the team.
I really like the idea of adding a Slack channel to the problem statement. I always have questions, and channeling them through a recruiter via email is a pain.
A lot of take home problems are very unrealistic, so it's difficult to make the trade-offs that you'd normally make when trying to ship software. Being able to ask someone on the team what their intentions are would be super helpful.
My pet peeve is front-end tests that ask you to write maintainable code, but you're not allowed to use any libraries. So instead you have to create your own tiny MVC/templating/helper libraries just for the project - but obviously they won't be as good as something you've spent more than 4 hours on.
A lot of take home problems are very unrealistic, so it's difficult to make the trade-offs that you'd normally make when trying to ship software. Being able to ask someone on the team what their intentions are would be super helpful.
My pet peeve is front-end tests that ask you to write maintainable code, but you're not allowed to use any libraries. So instead you have to create your own tiny MVC/templating/helper libraries just for the project - but obviously they won't be as good as something you've spent more than 4 hours on.
Unless the task was to write an MVC or helper library I'd be concerned the candidate wasn't using a library of some sort. It's really interesting getting a sense of where someone spent their time on a project - asking the candidate that question has been the turning point in the discussion because it a.) tells you if they actually wrote the code and b.) gives you a closeup view on how the person thinks
The take-home/programming tests in general are just as bad as interview questions. You're asking senior level engineers to take 3-5 hours (at what? 100-150$ hr bill-rate) to see if they can code?
Do you like them? Check. Do you feel like they are telling the truth? Check. If you use the 'try before you buy' method: You can easily see if they can handle the job/handle the pressure.
Also, you probably aren't the only company/avenue they are applying towards. All of the calls/tests add up.
The question is though, since companies are already using take-home tests/interview questions... can anyone confirm that these methods are more effective than a 'try before you by/contract to hire' position?
Do you like them? Check. Do you feel like they are telling the truth? Check. If you use the 'try before you buy' method: You can easily see if they can handle the job/handle the pressure.
Also, you probably aren't the only company/avenue they are applying towards. All of the calls/tests add up.
The question is though, since companies are already using take-home tests/interview questions... can anyone confirm that these methods are more effective than a 'try before you by/contract to hire' position?
Only desperate people do these tests. I know this first hand because I've been given a handful of them and can never find the motivation to complete them.
I'm not particularly a fan of take-home tests. But comparing that to mumbling at a white-board, while a software engineer is pretending to be an expert psychologist at analysing your ~train of thought~, and unfortunately, more often than not, someone in the board is only paying attention at how much of an WASP male you are or pass by it, where you need to do a task that you only repeat at interviews for a job that will not require that...compared to that, give me take home interviews any day of my life. Yes they are free work, but at least they are free work related to actual work. Live coding in the whiteboard performances are not part of the work. And the stress argument (that programmers must be able to deliver under stress) there are categories of stress, social anxiety of a live performance is not the stress we deal at our works. Sometimes I do good whiteboard coding interviews, sometimes I do bad whiteboard coding interviews, none of the situations I felt the merit was of the code or my knowledge of the subject, it is not a proof of how well a programmer does his job, is a somewhat related way of knowing if somebody knows who to code something and if he does not feel too nervous to code on a whiteboard instead of a computer to a bunch of strangers testing him. If you add that to the fact that some interviewers like be to randomly arrogant, you're missing some really nice but perhaps shy coders.
This is why we use Take-Home tests here at deezer.
The part we don't have got right yet is how to manage the length of such an exercise. Finding a problem where you have to both demonstrate how you handle several common issues and that is not too long is pretty hard.
I actually did a similar thing as an ops guy (debugging systems as described in some questions) and really liked the concept. On the other hand I work on mathematical riddles for fun so maybe I'm the core target for those.
It doesn't replace face to face interviews but I can see it as a good example of "how would face this real life situation you would encounter if you got the job?"
It doesn't replace face to face interviews but I can see it as a good example of "how would face this real life situation you would encounter if you got the job?"
Take home tests are the worst. Company says take home test will take 3 hours to complete. They never do. Schedule 2x or 3x the estimate. Especially if you want to impress the reviewer.
You send it over, then the company says no or yes, only to move to new stage.
In the worst case you ruined your weekend and received a no. But the company just took 10 minutes to arbitrarily reject your application.
From the company's perspective:
I've seen applicants receive friends'/roommates'/spouse's help on take home tests. Not a good indicator at all even with a glowing submission.