Show HN: Takehome.io – Time-limited coding challenges for interviews via Git(takehome.io)
takehome.io
Show HN: Takehome.io – Time-limited coding challenges for interviews via Git
https://takehome.io/
109 comments
I use Codility for code screening: https://www.codility.com/
In most of the Europe take home exams are common during the interview process, but your online test should be 1h (at most 2h). If you make it longer, very few ppl will invest that much time in competitive job market such as software engineering.
Take home tests are way harder in the hottest market such as San Francisco Bay Area. A lot of candidates will simply drop out of your pipeline if you send them. IMO there are good for fresh grads, junior candidates or candidates that would you otherwise reject in resume screening. There are hidden gems, often in underrepresented minority that simple don't have anything impressive on their resume yet. Being able to identify them can be your competitive advantage.
In most of the Europe take home exams are common during the interview process, but your online test should be 1h (at most 2h). If you make it longer, very few ppl will invest that much time in competitive job market such as software engineering.
Take home tests are way harder in the hottest market such as San Francisco Bay Area. A lot of candidates will simply drop out of your pipeline if you send them. IMO there are good for fresh grads, junior candidates or candidates that would you otherwise reject in resume screening. There are hidden gems, often in underrepresented minority that simple don't have anything impressive on their resume yet. Being able to identify them can be your competitive advantage.
I've used TestDome in the past, http://www.testdome.com.
The tests are tidy and the in-browser IDE is not bad.
The tests are tidy and the in-browser IDE is not bad.
I just completed the task in under 10 minutes for the example socialite one. It would be neat if the candidate could mark the repo as finished, so that you don't have to wait the whole time before it is over. Cool idea overall!
Thanks, that's a great idea. I'll add it to my burndown.
We send much smaller coding tests using https://remoteinterview.io/. Take-homes require way too much time commitment from the candidate side. We have tried them but the candidates would never finish.
With coding tests, we isolate a smaller problem and send that as a fizzbuzz screener. This scales well as solutions are checked automatically.
With coding tests, we isolate a smaller problem and send that as a fizzbuzz screener. This scales well as solutions are checked automatically.
I like the trend towards tools like this helping with the software engineering hiring process. I know that these things can be points of contention in the community, but in general the problem of reducing the time commitment from the hiring team while still providing the candidate with a low-friction experience that produces a meaningful expression of capability isn't going away any time soon.
Probably should offer a review service and coding challenges too! A company that is capable of reviewing and making code challenges surely has the competence to set up a git server, but there's plenty of companies that can't or don't have time, so I definitely think there's a market.
A service where I can `git clone` a repo and that is what starts the timer for the coding challenge is a great idea.
When I was recently job hunting, I had a bad experience with a company that wanted me to take a timed coding challenge where I would clone a repo and then submit a pull request with my solution. What started the timer then was them opening the repo to me, not me accessing it. This required us to agree on a time for me to take the challenge. Long story short, the recruiter proved incapable of opening the repo at the times we agreed on.
Avoiding that need for coordination and chance for the recruiter to mess things up is a nice benefit of your service.
When I was recently job hunting, I had a bad experience with a company that wanted me to take a timed coding challenge where I would clone a repo and then submit a pull request with my solution. What started the timer then was them opening the repo to me, not me accessing it. This required us to agree on a time for me to take the challenge. Long story short, the recruiter proved incapable of opening the repo at the times we agreed on.
Avoiding that need for coordination and chance for the recruiter to mess things up is a nice benefit of your service.
This is too expensive and it doesn't even let me take the challenges myself without paying $20AUD, even though is says "Free Trial".
Also, it's basically just 2 forms and some scripts.
Also, it's basically just 2 forms and some scripts.
It definitely _should_ allow you to challenge your own email address without any charge. If you're still having trouble and you'd like to test it out, please drop me a line on [email protected] and we'll figure it out. The trial has worked great for everyone else so it sounds like a juicy bug I'd love to hunt if it's not working for you.
I think this looks really cool. I like the ability to send multiple challenges to a candidate. I also like that as a side effect, it tests a candidates proficiency with Git.
I would choose a large time window- 4-8 hours or so. That way it ensures there is plenty of time to complete the challenge but also that it gets done in one day, like a normal workday.
I would choose a large time window- 4-8 hours or so. That way it ensures there is plenty of time to complete the challenge but also that it gets done in one day, like a normal workday.
By proficiency you mean clone and push?
Clone, add, commit messages, push, ignoring dependencies, writing an instructive README.md, etc. All basic skills you'd expect a developer to have.
>All basic skills you'd expect a developer to have.
Yeah, if they've used git before. Otherwise they'll need at least a solid 30 minutes to figure out the most basic git usage pattern.
Do you really care about what source control system a person has experience in? Why? Any competent developer can pick it up quickly enough.
Yeah, if they've used git before. Otherwise they'll need at least a solid 30 minutes to figure out the most basic git usage pattern.
Do you really care about what source control system a person has experience in? Why? Any competent developer can pick it up quickly enough.
Hardly you can call that proficiency. All they have to do here is to clone and push. Sorry but I tend to test git knowledge more in depth, maybe this level is enough for other projects
this looks very cool - the single biggest thing that makes or breaks this space is the strength of the integrated applicant tracking system (ATS).
Do I need to use another ATS ? then my team will just give up on this. If you have a limited ATS in there (with stuff like emailing from the dashboard, reading replies from the dashboard), then this is killer
Do I need to use another ATS ? then my team will just give up on this. If you have a limited ATS in there (with stuff like emailing from the dashboard, reading replies from the dashboard), then this is killer
Currently no ATS at all. I've specifically kept it focused on doing a single thing well. I appreciate the feedback, though. I'll consider incorporating something along those lines.
This is a great idea. I could see becoming a customer of yours if you added:
- payment in USD
- Ability to provide a post-receive hook to run validation / tests / benchmarks, and see results with the notification of submission. I appreciate that doing this in secure fashion is a nontrivial effort.
Thanks! Webhooks to implement that kind of post-receive functionality are on the TODO list. If you like, get in touch at [email protected] and I'll let you know once they're done.
> payment in USD
Honest question: why? Can't your card / bank / PayPal do the conversion for you? (I say this as a Canadian who often pays USD with my Canadian card / bank account).
Honest question: why? Can't your card / bank / PayPal do the conversion for you? (I say this as a Canadian who often pays USD with my Canadian card / bank account).
The bigger issue for me is pricing in USD, and less about the currency in which payment is collected. I'm in the UK, and while I can generally work out a rough exchange rate for GBP to USD and EUR in my head, I've no idea what the current AUD rate is. Making me check is a small but real barrier.
The idea of "take home tests" for an interview is utterly bizarre to me. Thankfully I've never heard of it my parts, seems to be a mainly US thing?
I think most jobs I applied to out of college had them. It is a good way to vet if someone can code before bringing them in for an interview and wasting a bunch of peoples' time. At my company we're currently struggling with not doing timed tests but not wasting too much of a candidate's time.
We hire a lot straight out of college, which I think is probably where this type of test is more common.
We hire a lot straight out of college, which I think is probably where this type of test is more common.
There were two jobs I was curious about that sent me to hackerrank.com with a six hour test and a twenty four hour test, respectively, in Amsterdam and Barcelona.
This is spreading like a plague in Europe as well.
Integrate the payment mechanism.
i will say that I like your pricing model. Free to try it out yourself and only $20 (or whatever the currency rate is in AUS) for each time you use it with no monthly fee.
Personally this is how most online services should be, pay per use.
Personally this is how most online services should be, pay per use.
I don’t like the timeboxing aspect. It’s putting pressure on the candidate. I don’t know why the website says there is less pressure when it’s time limited. Do you have a study or other evidence to support this claim?
For code exercises I set when recruiting I send a pdf out and then expect a zip file or link to GitHub or bitbucket with the solution. I might give an advisory along the lines of this shouldn’t take more than half a day. But if the candidate screws up and starts again that’s ok. Mistakes and wrong turns are taken all the time in the real world. It works very well. Simple. No complications. And free.
For code exercises I set when recruiting I send a pdf out and then expect a zip file or link to GitHub or bitbucket with the solution. I might give an advisory along the lines of this shouldn’t take more than half a day. But if the candidate screws up and starts again that’s ok. Mistakes and wrong turns are taken all the time in the real world. It works very well. Simple. No complications. And free.
Hey, thanks for the feedback.
The idea is that the candidate knows they can do the best they can within the time limit, and then it's done. No agonizing over whether it's good enough or they should spend another few hours. No imagining how many hours and days other candidates for the same role might be putting in, wondering if they're going to stack up.
I've used this approach with around 100 candidates hiring for various roles in my own business and for clients. The response I've gotten from candidates has been overwhelmingly positive.
The problem I have with the traditional "don't spend more than about X time" approach is that it's not a level playing field. The responses to a challenge exist in relative terms, not absolute. It doesn't matter whether your response was great or not. It matters whether your response was in the top 5 of all the other candidates responding. It follows that you probably need to spend at least as much time, if not more, than all the other applicants for the role. How much time the other candidates _might_ be spending is basically unbounded, leading candidates to spend vastly too much time working on a challenge and then lying about it.
If the candidate screws up a timed challenge that's also totally okay. It'll be a great conversation to have in the follow-up interview. "I started down this road, but if I was doing it again I'd do X, Y and Z because of A, B and C."
Incidentally, I'd suggest that half a day is way too long. Especially if that half a day isn't actually enforced. When I'm hiring I usually set challenges of about 40 minutes, an hour and a half max. I'm not looking for a product that's ready to ship to production. I'm after a sense of whether the candidate can actually interpret a problem then string some code together, and something meaningful we can discuss in the next interview.
The idea is that the candidate knows they can do the best they can within the time limit, and then it's done. No agonizing over whether it's good enough or they should spend another few hours. No imagining how many hours and days other candidates for the same role might be putting in, wondering if they're going to stack up.
I've used this approach with around 100 candidates hiring for various roles in my own business and for clients. The response I've gotten from candidates has been overwhelmingly positive.
The problem I have with the traditional "don't spend more than about X time" approach is that it's not a level playing field. The responses to a challenge exist in relative terms, not absolute. It doesn't matter whether your response was great or not. It matters whether your response was in the top 5 of all the other candidates responding. It follows that you probably need to spend at least as much time, if not more, than all the other applicants for the role. How much time the other candidates _might_ be spending is basically unbounded, leading candidates to spend vastly too much time working on a challenge and then lying about it.
If the candidate screws up a timed challenge that's also totally okay. It'll be a great conversation to have in the follow-up interview. "I started down this road, but if I was doing it again I'd do X, Y and Z because of A, B and C."
Incidentally, I'd suggest that half a day is way too long. Especially if that half a day isn't actually enforced. When I'm hiring I usually set challenges of about 40 minutes, an hour and a half max. I'm not looking for a product that's ready to ship to production. I'm after a sense of whether the candidate can actually interpret a problem then string some code together, and something meaningful we can discuss in the next interview.
All this does is compound the stress. Instead of "agonizing" over whether it's good enough, now a candidate gets to agonize over 1) is it good enough, 2) why didn't I just push the updated code 1 minute earlier before the cut off time! and 3) sh^t I should have put more comments in to explain where I was going with that!
>The problem I have with the traditional "don't spend more than about X time" approach is that it's not a level playing field.
No, it's not. And that's actually fine. I don't care if a candidate took five minutes, or five hours, or five days! In fact, the response time can tell you something about a candidate. Don't take that away from me as a hiring manager. I've actually never asked or been asked how much time I spent on a challenge. It's not an issue to solve!
>If the candidate screws up a timed challenge that's also totally okay.
What follow up interview? They fu^ked up the challenge. You really think a hiring drone is going to proceed that candidate when they have 10 others who did ok? That goes completely against what this sort of canned test is all about; getting past a hurdle. If hiring managers were going to interview all candidates who took the test and have a nice chat about their code, the test is completely redundant. They'll just ask for a code sample to talk about. Saves everyone the time and the money!
All in all, 40 minutes is way too short and artificial cut offs are a nasty way to try and "level the playing field". In fact, you're not leveling it at all! Just making sure candidates who may be taking a bit longer to get into the swing of things that day are discounted. Heck, if the candidate has simple network issues they're screwed!
>The problem I have with the traditional "don't spend more than about X time" approach is that it's not a level playing field.
No, it's not. And that's actually fine. I don't care if a candidate took five minutes, or five hours, or five days! In fact, the response time can tell you something about a candidate. Don't take that away from me as a hiring manager. I've actually never asked or been asked how much time I spent on a challenge. It's not an issue to solve!
>If the candidate screws up a timed challenge that's also totally okay.
What follow up interview? They fu^ked up the challenge. You really think a hiring drone is going to proceed that candidate when they have 10 others who did ok? That goes completely against what this sort of canned test is all about; getting past a hurdle. If hiring managers were going to interview all candidates who took the test and have a nice chat about their code, the test is completely redundant. They'll just ask for a code sample to talk about. Saves everyone the time and the money!
All in all, 40 minutes is way too short and artificial cut offs are a nasty way to try and "level the playing field". In fact, you're not leveling it at all! Just making sure candidates who may be taking a bit longer to get into the swing of things that day are discounted. Heck, if the candidate has simple network issues they're screwed!
Hi David,
I like the idea a lot, but I think it is up to the team and hiring manager and not the tool the decide whether to use a time box or not. I haven't tried the tool yet so I'm only speculating here, but as a potential user from the team / hiring manager side I would expect I could disable the time boxing completely if I wanted.
Cheers!
I like the idea a lot, but I think it is up to the team and hiring manager and not the tool the decide whether to use a time box or not. I haven't tried the tool yet so I'm only speculating here, but as a potential user from the team / hiring manager side I would expect I could disable the time boxing completely if I wanted.
Cheers!
If you’re setting deadlines of forty minutes then I suggest you’re losing good candidates, who like to assess a problem and think through a solution first. It’s just creating pressure. Unless your code exercises are takes on these fizzbuzz exercises? I.e. trivial.
From my experience (and I have a lot) you can tell a good developer by their deliverable. Even better if you see the code developing through multiple commits. Some developers might take a long time, but if it’s bad it’s just bad. I would worry that if you only set 40 - 90 minutes exercises you aren’t giving enough time for the candidate to differentiate themselves at all.
>Incidentally, I'd suggest that half a day is way too long. Especially if that half a day isn't actually enforced. When I'm hiring I usually set challenges of about 40 minutes, an hour and a half max. I'm not looking for a product that's ready to ship to production. I'm after a sense of whether the candidate can actually interpret a problem then string some code together, and something meaningful we can discuss in the next interview.
While I agree with what you said about leveling the playing field, I find 40 minutes sound awfully short. In university I noticed I was doing much better on 3h exams than on 1.5 hour exams, often only taking 1.5h for the 3h exam, even though it had more material. I couldn't relax enough in the short exam to do my best, even though the time was long enough for a twice as long exam.
While I agree with what you said about leveling the playing field, I find 40 minutes sound awfully short. In university I noticed I was doing much better on 3h exams than on 1.5 hour exams, often only taking 1.5h for the 3h exam, even though it had more material. I couldn't relax enough in the short exam to do my best, even though the time was long enough for a twice as long exam.
> In university I noticed I was doing much better on 3h exams than on 1.5 hour exams
Same here. I was the person who would do poorly on the 15 min physics quizzes we had in college just to get the highest grade in the class on the final, only because I had more time (half the answers on my quizzes would be blank just because I didn’t get to them). In high school, I would barely squeak by on the multiple choice AMC to then do fantastic on the 3 hour AIME.
But I clearly knew the material, and I knew it very well, so are you going to penalize me for being slightly slower on a test? As an employee, I complete my real world tasks just as fast as everyone else, so why should a time constraint on a high pressure exam relate to the real world productivity and profitability of an employee? I would argue that it doesn’t—at all.
Same here. I was the person who would do poorly on the 15 min physics quizzes we had in college just to get the highest grade in the class on the final, only because I had more time (half the answers on my quizzes would be blank just because I didn’t get to them). In high school, I would barely squeak by on the multiple choice AMC to then do fantastic on the 3 hour AIME.
But I clearly knew the material, and I knew it very well, so are you going to penalize me for being slightly slower on a test? As an employee, I complete my real world tasks just as fast as everyone else, so why should a time constraint on a high pressure exam relate to the real world productivity and profitability of an employee? I would argue that it doesn’t—at all.
But then again, a candidate that puts in way more effort is also a strong signal for future performance.
I suppose it depends on the employer, does he prefer candidates that put in more effort but might be less talented, or very talented but might not have much resolve (unproven at least).
I suppose it depends on the employer, does he prefer candidates that put in more effort but might be less talented, or very talented but might not have much resolve (unproven at least).
If I'm given a time limit on an assignment, I'm going to take the time allotted and it will reflect in my (admittedly easily faked) git history.
If the employer is being honest, then we're all good. But if they're going to go with the candidate that puts in 5x the time allotted, then I clearly don't want to work for them either.
It's an obvious red flag IMO. If they ask for n hours in the interview but expect more than that, then they're likely to do the same thing when you're on the job. Don't be surprised if a few months down the line you're working 12 hour days 6 days a week or you're an underperformer and the first to get cut.
If the employer is being honest, then we're all good. But if they're going to go with the candidate that puts in 5x the time allotted, then I clearly don't want to work for them either.
It's an obvious red flag IMO. If they ask for n hours in the interview but expect more than that, then they're likely to do the same thing when you're on the job. Don't be surprised if a few months down the line you're working 12 hour days 6 days a week or you're an underperformer and the first to get cut.
How often in their day job will someone be under artificial time pressure to solve a task and get it to a point where it's "good enough"?
You're actively removing the possibility of finding out who would think "meh thats good enough I guess" is the effort needed for production code and people who would go the extra step of spending time analysing and fully solving the problem.
2 people with equally average solutions...would one have done more given the chance? Both? Neither? How can we find out? You've added uncertainty to the process rather than solved any problem.
You're actively removing the possibility of finding out who would think "meh thats good enough I guess" is the effort needed for production code and people who would go the extra step of spending time analysing and fully solving the problem.
2 people with equally average solutions...would one have done more given the chance? Both? Neither? How can we find out? You've added uncertainty to the process rather than solved any problem.
> How often in their day job will someone be under artificial time pressure to solve a task and get it to a point where it's "good enough"?
There are quite a lot of software jobs like this. Maybe the time pressure is more 2 weeks instead of 4 hours, but deadlines are a real thing.
There are quite a lot of software jobs like this. Maybe the time pressure is more 2 weeks instead of 4 hours, but deadlines are a real thing.
Two weeks is decidedly not 4 hours, no matter how you try to relate them. I could write the Falcon Heavy takeoff sequence in two weeks. (joking, of course, but the compression factor of 4 hours over two weeks is overwhelmingly and obviously higher)
> How often in their day job will someone be under artificial time pressure to solve a task and get it to a point where it's "good enough"?
In my job, pretty much every week. Many of my tasks are "implement feature X to a passable amount in 2 days", and then it goes to someone to use and get feedback on immediately. My team are mostly the same.
In my job, pretty much every week. Many of my tasks are "implement feature X to a passable amount in 2 days", and then it goes to someone to use and get feedback on immediately. My team are mostly the same.
This sounds to me like the nature of every job. Secondly, the interview process has never, to my knowledge, reflected on the job performance, and I'd suggest it shouldn't.
Often enough someone will be under real time pressure to solve a task, right?
Many times, markets and business dictate time pressure. So yes, this is very real in startups at least. Like for instance, an inevitable market development might want you to ship code by the next week, however improbable or impossible. I know I'll get a lot of flak for saying this, but that's ok. I've experienced it first hand. In an early stage startup, when faced with the choice of losing revenue/delivering on time, you'll choose the latter. That means your shipping slot is fixed and so are your features. You'll just have to slog it out or work real smart. Even if you cut features, you'll still realize there's a lot to be accomplished in a short span of time. This is mostly a one off thing. If such events repeat endlessly, that is classic bad management. So you'll need to deliver on time, sometimes :).
So, I guess that companies which administer such time based tests foresee at least one such situation for that role. I'm just speculating.
So, I guess that companies which administer such time based tests foresee at least one such situation for that role. I'm just speculating.
That's more indicative of poor management and planning on the company's side than reflecting on the performance of the candidate if the time pressure is a regular occurrence.
Few people perform well under pressure. Do you want to test what is their mediocre performance under time limits like on a school test or what they are really capable of?
With this approach you will hire people able to churn out crappy but "good enough" code in the limited time. Not people who actually can do quality work.
Few people perform well under pressure. Do you want to test what is their mediocre performance under time limits like on a school test or what they are really capable of?
With this approach you will hire people able to churn out crappy but "good enough" code in the limited time. Not people who actually can do quality work.
No real company has perfect management and a lot of them have poor planning.
> Few people perform well under pressure.
Yes, but majority of jobs have some level of time pressure. That being said, there is such a thing as reasonable deadline.
> With this approach you will hire people able to churn out crappy but "good enough" code in the limited time. Not people who actually can do quality work.
Inability to perform under pressure does not imply high skill without pressure.
> Few people perform well under pressure.
Yes, but majority of jobs have some level of time pressure. That being said, there is such a thing as reasonable deadline.
> With this approach you will hire people able to churn out crappy but "good enough" code in the limited time. Not people who actually can do quality work.
Inability to perform under pressure does not imply high skill without pressure.
> Inability to perform under pressure does not imply high skill without pressure.
Ability to churn out crap in a time limit doesn't imply competence neither.
Ability to churn out crap in a time limit doesn't imply competence neither.
There's never enough time or resources to do everything you want to do. Believe me; I've gone from one extreme to the other in company and team size. For any but the most mechanical of development tasks (e.g. CRUD apps), it's very common for a developer to make an estimate, realize that it was too optimistic, and then have to reduce the scope. So it's valuable for a developer to spend some time up front thinking about the design, break it down into tasks, prioritize them, and then implement the highest-priority part first. Then, even if they have to cut tasks, they can deliver something of value without having to do shoddy work.
I do think an interviewer should be completely up-front about what they want out of a candidate if they give a coding challenge that's too ambitious to implement perfectly within the time limit.
I do think an interviewer should be completely up-front about what they want out of a candidate if they give a coding challenge that's too ambitious to implement perfectly within the time limit.
> When I'm hiring I usually set challenges of about 40 minutes, an hour and a half max.
No. Forty minutes is far too little time. Many people take 10-15 minutes to even get settled into coding. I can't imagine a meaningful, applied, realistic exercise that would take 40 minutes, unless you're wanting basic data structures like a linked list, or a basic sorting algorithm, or something.
You're missing a lot of good people that way, but I guess that's just collateral damage in the quest for the perfect hire, right?
No. Forty minutes is far too little time. Many people take 10-15 minutes to even get settled into coding. I can't imagine a meaningful, applied, realistic exercise that would take 40 minutes, unless you're wanting basic data structures like a linked list, or a basic sorting algorithm, or something.
You're missing a lot of good people that way, but I guess that's just collateral damage in the quest for the perfect hire, right?
Except your path is the one that has lots of long, burdensome take home tests. Some people have time commitments and/or other jobs to apply to but I guess those people are just collateral damage in your quest for the perfect hire, right?
Is it possible that there could be a happy medium?
An 1-1.5 hour test is a happy medium.
I would agree that 1.5 hour test is a happy medium. I'd prefer 2 hours.
I am guessing fizzbuzz or something similar. I was applied for a job and got asked to do an n queens programming task. It was for a commodity trading firm. I told them no thanks. When I set code exercises I craft them myself and make them appropriate to the job on offer.
Robust answer and well reasoned argument. I agree, and would go further to say that less pressure and more satisfaction has been had with time restricted homework in my experience as an interviewee.
You have your reasons, but I'll walk from a time limited take-home, just as I'll walk from a screen captured one.
Both methods are ways to tilt the control and convenience back in the favour of the company - defeating the whole purpose of doing the work at home.
With all due respect, but attempts to optimize the interview process invariably end up doing this, due to the priorities of the paying parties (ie - the companies) - and always at the cost of the interviewers. No thanks.
Both methods are ways to tilt the control and convenience back in the favour of the company - defeating the whole purpose of doing the work at home.
With all due respect, but attempts to optimize the interview process invariably end up doing this, due to the priorities of the paying parties (ie - the companies) - and always at the cost of the interviewers. No thanks.
I worry you might be projecting your experience onto the experiences of others. It sounds like you’ve been stressed out about having an unlimited time slot in the past.
But lots of people (myself included) get debilitatingly stressed out when an arbitrary clock is put in front of your face. Honestly, my recommendation to any company that gives timed challenges is to... not.
What if you gave the option to not time challenges? And maybe didn’t make it seem like having untimed challenges are inherently more stressful all the time.
But lots of people (myself included) get debilitatingly stressed out when an arbitrary clock is put in front of your face. Honestly, my recommendation to any company that gives timed challenges is to... not.
What if you gave the option to not time challenges? And maybe didn’t make it seem like having untimed challenges are inherently more stressful all the time.
I mean, it's strictly better than the typical code interview, which is timed too. And the reality today is that the majority of people have to be able to deal with a ~45min code interview to find employment in software.
I recently interviewed for my next role. Not a single company was doing the in-person timed code interviews. Where there were code tests, they were all take home and had no time limit beyond "when do you think you can get this back to us by?". If you need a time limit, ask the candidate for their estimate. It has the added benefit of telling you whether the candidate can assess their own schedule in relation to your code test, while not screwing the candidate over.
I think your approach is reasonable. But I'm curious, why provide the challenge in PDF rather than HTML? Unless your PDF generator can create a tagged PDF, that's going to be less accessible for blind people to blind people than HTML, .docx or .odt, or even plain text. I don't mean to guilt-trip here; I figure you weren't aware of that problem. Is there a reason why PDF is better? For example, do your challenges tend to include graphics or complex math notation?
In the past when doing timeboxed, we gave the candidate a weekend. The task was too much work for a weekend, but that's part of the job. Choosing to prioritize and what to prioritize is also an important skill. In our case: if you were going for the perfect code abstraction but didn't deliver any business value in the form of a working product, that says something about the candidate and their performance in a team.
Do you tend to give a two day deadline in the workplace and then expect the deadline to not be met?
We were open and honest about it being too much. The pattern that emerged is that it allowed us to see the candidates' strengths. Some didn't tackle the backend part: that was not their forte (and we hired them and sent them off on a course).
Also, in any IT job I've worked there's generally more business requirements/wishes/etc than can be finished. That's a fact of life. It helps to have team players that have are independent and give some push back where needed, as well as realistic deadlines. There were some that handled the deadline badly, that didn't help their case.
(but apparently, many believe giving someone a task to do is not done)
I guess what I'm saying is the expected reaction to this in the work place would be to trim the requirements and set expectations before you ever start work. If that's part of the test, cool, but it seems like that wouldn't be something a candidate would assume heh.
I hope you communicated that clearly beforehand?
This! I've done more than one code challenge in my time that was secretly testing for how far you get on a task that is underbudgeted on time. Your best applicants are going to spend too much of their own (unpaid) time finishing it and be annoyed at you before you tell them how clever you are for testing their ability to prioritize...or they're just going to not bother.
I would assume this is how the company will treat me after being hire.
Far too many companies forget that they are also being tested and interviewed. The sort of stupid "let's secretly test whether the candidate can figure out we're screwing with them" kind of thing lets good candidates know one thing: the company is toxic and won't treat them like a human. This is the exact sort of behavior to run away from.
Yeah because most peoples biggest priority going into in a technical interview process is focusing on good code. If they want a candidate to "MVP" their coding challenge for business value reasons, they should make that very clear. It's not like it takes MORE skill to write hacky code.
Definitely
Do you want crappy code that get's harder and harder to extend and maintain? Because that's how you get that.
Correct, but time to market has it's value too. Also: mobile dev so Apple forces a rewrite every couple years anyway (on both design and tech fronts)
I like the gist of this (I think take-homes are the best way to evaluate a candidate), but making it time limited seems weird. You're basically taking a test that you haven't studied for -- that doesn't seem fair (or fun).
What is an interview if not a test? The code challenge portion of an interview process is a test that you've been studying for your entire career.
As for fairness, I'd argue that timing it is the only way to ensure it. This way you know that the playing field is level between you and the other candidates applying for the role.
To fun, I'd say that the least fun thing I can think of in this context is spending days wondering whether I had "done enough yet" in order to make my submission stand out ahead of the other applicants. Much more fun to know that as soon as that clock ticks over, what's done is done and I can get on with my life for better or worse.
As for fairness, I'd argue that timing it is the only way to ensure it. This way you know that the playing field is level between you and the other candidates applying for the role.
To fun, I'd say that the least fun thing I can think of in this context is spending days wondering whether I had "done enough yet" in order to make my submission stand out ahead of the other applicants. Much more fun to know that as soon as that clock ticks over, what's done is done and I can get on with my life for better or worse.
An interview is not a test. An interview is a two-way conversation to determine fit for both parties. No test can possibly match human intuition.
In my experience on both sides of the desk, a more holistic approach yields better results in the long run. A resume scan, then a review of presented code, then a phone screen to determine communications ability and check some boxes, then a short series of in-person conversations with the team to determine "can we work together?"
In my experience on both sides of the desk, a more holistic approach yields better results in the long run. A resume scan, then a review of presented code, then a phone screen to determine communications ability and check some boxes, then a short series of in-person conversations with the team to determine "can we work together?"
How does your approach differ from the sites being already used by companies for online coding test? They do the same thing, you login during some fixed duration(24-48hr) in the weekend, and you do the coding challenges(they have a web interface to compile your code) within the alloted time.
I think the main difference is that those sites are geared towards outputting a ranking or score of some time. Takehome is geared at giving you a way to qualitatively assess the skills of each candidate and give you something meaningful to discuss with them in a follow-up interview.
Also, all of the existing sites I've seen have one or more of the following problems:
* Make you use their editor * Make you install their software on your computer * Make you use some specific language * Make you use their set of challenges
Takehome uses nothing but git. Whatever you put in the repository and how you get it there is up to you.
That holds true both for the candidate and the challenger. Takehome encourages challengers to create their own task matching the kind of work/language/framework/etc that's relevant to their team.
Also, all of the existing sites I've seen have one or more of the following problems:
* Make you use their editor * Make you install their software on your computer * Make you use some specific language * Make you use their set of challenges
Takehome uses nothing but git. Whatever you put in the repository and how you get it there is up to you.
That holds true both for the candidate and the challenger. Takehome encourages challengers to create their own task matching the kind of work/language/framework/etc that's relevant to their team.
If the goal is to not be stressful, why are we treating interviews like tests?
Actually, I wouldn’t mind as much if they gave hints at what the problem might be so I could study for it. A day to prepare for a one hour coding test, that might be fun.
This seems useful, and I get the value. I'm wondering how you balance exactly how much time to allot? Do you try to come up with an average estimate of time needed and then like 2x that to reduce time anxiety?
Thankyou! Any person using the site to hire candidates can set whatever time limit they like.
When I'm choosing a time I'm actually less worried about the complexity of the exercise than how much of a candidate's time it's reasonable to ask for. If the challenge is too complex to be done in 40 minutes or so, I make it less complex.
I also put language in the intro text like 'If you don't end up with a complete solution in that time, that's fine, just submit it anyway. We don't expect this to be enough time to write "The Most Perfect Thing".'
If you're curious, sign up and take a look at the "Socialite" sample challenge.
When I'm choosing a time I'm actually less worried about the complexity of the exercise than how much of a candidate's time it's reasonable to ask for. If the challenge is too complex to be done in 40 minutes or so, I make it less complex.
I also put language in the intro text like 'If you don't end up with a complete solution in that time, that's fine, just submit it anyway. We don't expect this to be enough time to write "The Most Perfect Thing".'
If you're curious, sign up and take a look at the "Socialite" sample challenge.
My rule of thumb was to decline any interview with "challenges" - works perfectly as a filter of perverted corporate climate.
In my experience, whether an interviewer gives a take-home challange has absolutely 0 correlation to the corporate climate. These things are usually decided on a whim by the interviewer, who has no effect on the nature of day-to-day work of an employee. If you ask them, they'll just say "I don't know, just sounded like a good idea. Don't see why not." So you're just filtering based on random noise.
Isn’t that survivorship bias? How do you know you didn’t pass up on some great opportunities? I’ve met at least some great companies that were a pleasure to work with, who did take homes.
I ended up working 100% remote and never looking back.
Sometimes I was taking (remotely) corporate interviews just to be sure I am not missing something, I don't do this anymore. Skype-ing guys who are talking about their great company culture from 2x2 m cubicles was sometimes fun though.
I'd love it if I could just hire people who had already done some work for me, or who had a really good Github portfolio. However demanding that rules out loads of other good candidates. I find the take home project to be the best alternative. I normally only give them to people that I expect to interview. I don't expect many to "fail", though it filters out the worst and identifies some stars. I find the conversation that I then have with them about it in the interview is the most useful part.
I wonder how would you do hiring for your fictional company? (Assuming you get 100s of shitty dev resumes when you post a job opening)
I am actually hiring for my contracting projects - I just have a general talk and ask for a piece of any recent code. And even this is just a ritual: only real project in your (not other company) real work environment shows.
Well, if a candidate whipped out code from their work project then I would not hire, as it demonstrates a severe lack of respect of their employer's intellectual property.
Of course, unless it was already on Github, etc.
[while I'm in the "there's no secrets in software" camp, there are still policies and laws to follow]
Of course, unless it was already on Github, etc.
[while I'm in the "there's no secrets in software" camp, there are still policies and laws to follow]
> any recent code
which is unlikely to be available if they aren't the owners of the IP.
So either you have to guess real lucky, or have a probation period of a week or two with any new hire. Wouldn't you say that's worse than a short take-home project (for the employee)?
which is unlikely to be available if they aren't the owners of the IP.
So either you have to guess real lucky, or have a probation period of a week or two with any new hire. Wouldn't you say that's worse than a short take-home project (for the employee)?
I am not saying I don't have problems with finding people, I am saying here that companies are creating even worse problems by their naive automated HR assumptions.
Honestly, I probably wouldn't hire anyone that doesn't have their own personal side projects. That alone shows ambition, enthusiasm for learning, and it will show you their capability. Otherwise, working for a reputable company? Just because someone does well on a dumb test, doesn't mean much at all.
Just to flesh out this hypothetical:
You've possibly eliminated a lot of adults who are 10+ years out of college and have family commitments.
You've possibly eliminated people with significant side-interests outside the realm of software. Writers, artists, musicians, marathon runners, activists, etc.
Just because someone doesn't have side projects doesn't mean that they don't have ambition, enthusiasm for learning, or other ways to show you they're perfectly capable.
You've possibly eliminated a lot of adults who are 10+ years out of college and have family commitments.
You've possibly eliminated people with significant side-interests outside the realm of software. Writers, artists, musicians, marathon runners, activists, etc.
Just because someone doesn't have side projects doesn't mean that they don't have ambition, enthusiasm for learning, or other ways to show you they're perfectly capable.
Obscure programming tests will also filter out all of your examples. I think finding someone that is genuinely interested in programming, more than just a 9-5, is a better hire than someone who studied "Cracking the Coding Interview". Also, if someone is super dedicated enough to run marathons, I would put them at the top of the list as well. All opinion of course!
The issue I take with this opinion as noted above, it selects for people who have available time. I am genuinely interested in programming, but I come home and spend time with my 4 month old son.
I think the "more than just a 9-5" idea is a toxic pattern which has been championed by the privileged. In short, by wanting something like this, you risk selecting for a fairly homogenous set of people.
I think you can write a good coding challenge which doesn't look like all the others (the socialite example in Takehome.io is particularly good because it scales with skill level – it's not a pass/fail, but a spectrum from just making it work, to showing you have thought about and dealt with an array of odd things that might crop up).
I think the "more than just a 9-5" idea is a toxic pattern which has been championed by the privileged. In short, by wanting something like this, you risk selecting for a fairly homogenous set of people.
I think you can write a good coding challenge which doesn't look like all the others (the socialite example in Takehome.io is particularly good because it scales with skill level – it's not a pass/fail, but a spectrum from just making it work, to showing you have thought about and dealt with an array of odd things that might crop up).
That's fine, and I think the nonsense testing that is popular with valley jobs is a toxic pattern as well. I'm sure there are reasonable coding challenges, but none of them are ever highlighted around here.
The biggest reason for this attitude stems from the immense amount of people who are getting into CS just for a paycheck. Sure, we're only working to get paid, but in my experience, these people have zero motivation outside of it. Endless hordes of bootcamp type people, who see six figure salaries, and can spew out all of the stuff they've memorized in a book. Without actually thinking about why or when. I'm not looking for "rockstars", "ninjas", "gurus" or whatever other nonsense words phrases use, but rather people who are genuinely interested in their field.
The biggest reason for this attitude stems from the immense amount of people who are getting into CS just for a paycheck. Sure, we're only working to get paid, but in my experience, these people have zero motivation outside of it. Endless hordes of bootcamp type people, who see six figure salaries, and can spew out all of the stuff they've memorized in a book. Without actually thinking about why or when. I'm not looking for "rockstars", "ninjas", "gurus" or whatever other nonsense words phrases use, but rather people who are genuinely interested in their field.
I'd suggest some care with this approach. I've seen more than one candidate say they have no side projects, but when prodded further, we figured out they do in fact have side projects, the candidates just didn't twig on that they were side projects.
For example, writing up a silly application to make a daily task easier. One candidate had their own tooling for time tracking. It wasn't much to look at but it was there and working for them. It wasn't on github, well commented or entirely complete. It was however a neat tool and well enough written. The candidate didn't really think about it as a project, just a little tool they wrote.
For example, writing up a silly application to make a daily task easier. One candidate had their own tooling for time tracking. It wasn't much to look at but it was there and working for them. It wasn't on github, well commented or entirely complete. It was however a neat tool and well enough written. The candidate didn't really think about it as a project, just a little tool they wrote.
I mean that's just a fault of the candidate. You should be presenting anything relevant to the job you're applying for.
Don't see the value of the service. It just sounds like a simple CRUD app where people can create "challenges" and send them out to potential candidates.
And for so little done by the service, I find the cost to be completely perverse.
And for so little done by the service, I find the cost to be completely perverse.
Yep, all it does is wait for the first `git push`, display the challenge text, then allow further pushes before the time limit and denies them after the time limit. A per candidate price quickly gets expensive.
The price is per-candidate to discourage companies from sending out more challenges than they have time to thoughtfully review. The expected use case is that you interview the candidate, then if you are both still interested you issue a takehome challenge, then you have another interview to discuss the challenge submission.
If you're issuing more than 5 challenges per role you fill something somewhere has probably gone awry. $100 per role hired is no more than a rounding error.
Also, the timer starts from the first `git clone` rather than the first push.
If you're issuing more than 5 challenges per role you fill something somewhere has probably gone awry. $100 per role hired is no more than a rounding error.
Also, the timer starts from the first `git clone` rather than the first push.
Your pricing seems rather paternalistic. Let people use your service how they want to!
Are sample questions provided? Offer good quality questions and a monthly rate for a certain number of candidates. In some organisations it's easier to approve a monthly expense than a per use expense.
You should probably check the user can git push before starting the timer.
Are sample questions provided? Offer good quality questions and a monthly rate for a certain number of candidates. In some organisations it's easier to approve a monthly expense than a per use expense.
You should probably check the user can git push before starting the timer.
I recently started a new job. Before I spoke to anyone, I was issued a Hackerrank test to complete. I'm not saying I like it, but the coding challenge is often near the beginning of the hiring funnel to weed people out before the investment of interviewing.
the hole product could be a job interview challenge ;)
IMHO trying to educate companies about their process is a sure way to failure.
Also doing interviews before technical challenge is a sure suicide for companies. In my experience to minimise the waste of time you send out quick and easy technical challenge (no more than 30 minutes!) and you filter out 90% of applicants. With remaining 10% you have a phone call and if you have more than X (magic number of 3 in my experience) of qualified applicants you send out more thorough tech challenge (no more than 2h long!).
Rinse and repeat until you get to X applicants which you will invite for face to face interview.
Truth is that almost no-one in tech is qualified to do face-to-face interviews. Either they are not on the level technically or dont have the communication and HR skills. Interview in person is mostly to see if you are not dealing with a weirdo and if that person will be a good fit for the team.
A side note, I tend to issue valuable response to every candidate. If you do that, they are very likely to re-apply, step up when someone drops out or recommend to friends. People tend to do/say the same dumm stuff so you can prepare those replies as well.
Also doing interviews before technical challenge is a sure suicide for companies. In my experience to minimise the waste of time you send out quick and easy technical challenge (no more than 30 minutes!) and you filter out 90% of applicants. With remaining 10% you have a phone call and if you have more than X (magic number of 3 in my experience) of qualified applicants you send out more thorough tech challenge (no more than 2h long!).
Rinse and repeat until you get to X applicants which you will invite for face to face interview.
Truth is that almost no-one in tech is qualified to do face-to-face interviews. Either they are not on the level technically or dont have the communication and HR skills. Interview in person is mostly to see if you are not dealing with a weirdo and if that person will be a good fit for the team.
A side note, I tend to issue valuable response to every candidate. If you do that, they are very likely to re-apply, step up when someone drops out or recommend to friends. People tend to do/say the same dumm stuff so you can prepare those replies as well.
I thought I would chip in on the pro-time limit side as we seem to be underrepresented so far. I certainly have been given challenges that supposedly take 6 hours but clearly take much longer than that. An entire weekend at least. I don't have that time and I declined but if it really was a 6 hour test they would have gotten an attempt from me. A time limit makes it fair for those with other commitments.
To take it all the way, I think the follow-up discussion should be done strictly in text, using some combination of IM and a web-based code review tool. So the interviewer would call out parts of the PR and comment on them through the code review tool, and the two could have more free-flowing discussion over IM. Again, this would be just like the way it's actually done on the job, at least in my dream all-remote environment.