The business of takehome assessments(careerfair.io)
careerfair.io
The business of takehome assessments
https://www.careerfair.io/takehome-assessments
61 comments
> The direction is always the same "spend max a couple hours!"
Interestingly, I always took restrictions on time seriously. When I was told I should spend max 3 hours, I stopped latest at 3 hours, often earlier and discussed shortcommings of the solution based on spent time in the interview.
That strategy is easier when you are not that interested in that particular job though.
Interestingly, I always took restrictions on time seriously. When I was told I should spend max 3 hours, I stopped latest at 3 hours, often earlier and discussed shortcommings of the solution based on spent time in the interview.
That strategy is easier when you are not that interested in that particular job though.
Every job I tried this approach on were immediate rejections. Only solutions that completely fulfilled the requirements, with lots of testing, etc. were accepted for next steps. IE. I've never seen anyone who actually meant it when they gave a time limit.
> The direction is always the same "spend max a couple hours!" but the understanding is clearly you must spend significantly more.
I am in Product now, but when I was in SRE, I got one like this. "We don't want you to spend more than 3-4 hours on this", where "this" was:
Build a log parser for streaming Apache CLF. The parser should:
Keep a rolling monitor of the top 10 requests, displaying their velocity in req/sec as a rolling average.
Display aggregates of visitor counts over the last hour and day.
Have high watermark alerts when ingress traffic as a whole, or to hotspot URLs hit thresholds, and then be able to de-alert when traffic dropped.
Scaffolding to deploy same.
Unit tests and documentation for same.
Ability to ensure URLs were safe, stripped of any GET parameters.
Not overly complicated, but you're not busting that out in 3 hours in any polished manner, if at all.
Allo
I am in Product now, but when I was in SRE, I got one like this. "We don't want you to spend more than 3-4 hours on this", where "this" was:
Build a log parser for streaming Apache CLF. The parser should:
Keep a rolling monitor of the top 10 requests, displaying their velocity in req/sec as a rolling average.
Display aggregates of visitor counts over the last hour and day.
Have high watermark alerts when ingress traffic as a whole, or to hotspot URLs hit thresholds, and then be able to de-alert when traffic dropped.
Scaffolding to deploy same.
Unit tests and documentation for same.
Ability to ensure URLs were safe, stripped of any GET parameters.
Not overly complicated, but you're not busting that out in 3 hours in any polished manner, if at all.
Allo
I got a similar one once, like a fully working mocked banking system with currency exchange in 3 hours.
I just didn't proceed with the interview after that, if the staff can't manage to plan how long a test takes, there's no chance that they can plan anything accurately in the company either.
I just didn't proceed with the interview after that, if the staff can't manage to plan how long a test takes, there's no chance that they can plan anything accurately in the company either.
>I’m 27 and single. There lies the biggest advantage of a takehome assessment for me: I get to spend more time on it than someone with a wife and two kids.
Counterpoint: those fresh out of school are likely to do the best in some "invert a binary tree" leetcode style interview, with that fresh in their memory. Those who are older (and thus more likely to have families) who actually have experience at the craft are most likely to be able to bang out an actual coding task quickly.
Counterpoint: those fresh out of school are likely to do the best in some "invert a binary tree" leetcode style interview, with that fresh in their memory. Those who are older (and thus more likely to have families) who actually have experience at the craft are most likely to be able to bang out an actual coding task quickly.
One of the major problems with take home tests is the rampant cheating.
It's been a long time since I've been on the job market - but even a decade ago, I had third-party recruiters who would forward me a take home test... and another successful candidate's solution, "for reference".
And even if candidate doesn't get external help, I'm extremely sceptical about time limits. Nothing stops someone spending 6 hours on a "2 hour" take home test, producing a better solution than someone who followed the time limit strictly.
It's pretty hard to extract a signal under such circumstances. Unless you're hiring for a job where a willingness to bend the rules to get results is desirable - used car sales, for example.
It's been a long time since I've been on the job market - but even a decade ago, I had third-party recruiters who would forward me a take home test... and another successful candidate's solution, "for reference".
And even if candidate doesn't get external help, I'm extremely sceptical about time limits. Nothing stops someone spending 6 hours on a "2 hour" take home test, producing a better solution than someone who followed the time limit strictly.
It's pretty hard to extract a signal under such circumstances. Unless you're hiring for a job where a willingness to bend the rules to get results is desirable - used car sales, for example.
> Nothing stops someone spending 6 hours on a "2 hour" take home test, producing a better solution than someone who followed the time limit strictly.
Unless you can ... demonstrate you only spent two hours. It's pretty rare you can do that, but... several years back I got a callback on an interview, and they forwarded me the 'coding test' portion. I got it around 1:30p, and I emailed them back the finished product around 3:45p, so.. a bit over two hours, including download and setup of their test environment.
I advanced further, but they ultimately chose someone else. I think I may have been a bit too senior/advanced for them, or.. maybe I was just a jerk in the interview, or... someone else was just a better fit? I'll never know.
I do know it was one of handful of interviews in the last 10+ years that was actually done well. On the technical side, they had a test git repo that actually worked - fork the repo, pull down, run the docker compose and everything just worked. I spoke to the woman who'd set it up and it was obvious she'd put a lot of time in to making it all as smooth as possible, and it was. I wanted to work there just because she'd put that much effort just in to 'a test'.
Unless you can ... demonstrate you only spent two hours. It's pretty rare you can do that, but... several years back I got a callback on an interview, and they forwarded me the 'coding test' portion. I got it around 1:30p, and I emailed them back the finished product around 3:45p, so.. a bit over two hours, including download and setup of their test environment.
I advanced further, but they ultimately chose someone else. I think I may have been a bit too senior/advanced for them, or.. maybe I was just a jerk in the interview, or... someone else was just a better fit? I'll never know.
I do know it was one of handful of interviews in the last 10+ years that was actually done well. On the technical side, they had a test git repo that actually worked - fork the repo, pull down, run the docker compose and everything just worked. I spoke to the woman who'd set it up and it was obvious she'd put a lot of time in to making it all as smooth as possible, and it was. I wanted to work there just because she'd put that much effort just in to 'a test'.
> maybe I was just a jerk in the interview
I will say that this is something you definitely should not be, especially if it's a larger corp. You should be political, passionate, approachable and friendly. It doesn't matter if you are more technically skilled. It might be even worse for interviewers if you are technically skilled and arrogant about it, because it would make their work lives worse. Even though I appreciate technical skill and passion, I wouldn't hire anyone who was actively arrogant or similar about it. I look to create a team that in theory would be supportive or empathetic towards low performers, although of course preferably we would find a solution to low performers or understand why they are performing low if possible, because ultimately I do want high performing team.
I will say that this is something you definitely should not be, especially if it's a larger corp. You should be political, passionate, approachable and friendly. It doesn't matter if you are more technically skilled. It might be even worse for interviewers if you are technically skilled and arrogant about it, because it would make their work lives worse. Even though I appreciate technical skill and passion, I wouldn't hire anyone who was actively arrogant or similar about it. I look to create a team that in theory would be supportive or empathetic towards low performers, although of course preferably we would find a solution to low performers or understand why they are performing low if possible, because ultimately I do want high performing team.
I certainly don't try to be... and I generally don't think I come across that way (rude, arrogant, etc) but I'm sure I've had my moments unintentionally. There's a limit to how much influence I can have on someone's perception, and I know I've said things in the past that have upset folks without meaning to, or just... knocked me out of the running for a particular technical point of view.
That you understand that another's perception can be different than yours puts you in a high tier in my book.
It's like most things can be done right - If anyone just cared to! And when I run into something done right, it's so refreshing.
But most things we interact with are done half-assedly, or are MVPs, or are deliberately made to a stupid spec, or don't prioritize the user, etc... there are a million ways to mess it up - usually by not even trying - and the result is draining.
But most things we interact with are done half-assedly, or are MVPs, or are deliberately made to a stupid spec, or don't prioritize the user, etc... there are a million ways to mess it up - usually by not even trying - and the result is draining.
Before I go any further, broadly speaking, I'm not in favour of take home tests, but not for the reasons you mention.
I did a stint of hiring for a software company over a decade ago where they did use a take home test. Now we did get the odd person cheat on it, but I can think of only a handful of occurrences where that had happened - at least in a way where it had meaningful impact - out of the probably hundreds of people we interviewed.
The take home test was simply there to figure out whether or not it was worth inviting an engineer in for interview, at which point we of course did an in person coding assessment. If you cheat at the take home test, what happens at the interview? You get found out is what happens, and it can lead to quite an awkward conversation. If you don't get found out perhaps it doesn't matter anyway because you've just got through a coding interview anyway. That's not to excuse cheating but to point out that it's perhaps only a risk in terms of technical skill if it's your sole point of assessment of those skills.
But, anyway, as I say: I don't much like take home tests for other reasons. And I don't much like platforms like Codility either (with these I found the signal to noise ratio to be particularly poor). The reason I don't like any of this stuff though is that, if you're hiring, most likely it disadvantages you as an employer versus other companies who don't put candidates through these tests.
Unless you work for a company where everyone wants to work you have to ask yourself this question: why would anyone willingly go through our burdensome and tedious selection process when they can get a job on a similar, or maybe even better, salary elsewhere much more easily?
I did a stint of hiring for a software company over a decade ago where they did use a take home test. Now we did get the odd person cheat on it, but I can think of only a handful of occurrences where that had happened - at least in a way where it had meaningful impact - out of the probably hundreds of people we interviewed.
The take home test was simply there to figure out whether or not it was worth inviting an engineer in for interview, at which point we of course did an in person coding assessment. If you cheat at the take home test, what happens at the interview? You get found out is what happens, and it can lead to quite an awkward conversation. If you don't get found out perhaps it doesn't matter anyway because you've just got through a coding interview anyway. That's not to excuse cheating but to point out that it's perhaps only a risk in terms of technical skill if it's your sole point of assessment of those skills.
But, anyway, as I say: I don't much like take home tests for other reasons. And I don't much like platforms like Codility either (with these I found the signal to noise ratio to be particularly poor). The reason I don't like any of this stuff though is that, if you're hiring, most likely it disadvantages you as an employer versus other companies who don't put candidates through these tests.
Unless you work for a company where everyone wants to work you have to ask yourself this question: why would anyone willingly go through our burdensome and tedious selection process when they can get a job on a similar, or maybe even better, salary elsewhere much more easily?
> It's pretty hard to extract a signal from that.
It's hard to extract a signal about prospective work quality from anything but a trusted referal or a probationary hire. It's not like there's a great alternative just being ignored.
Some orgs feel more comfortable assuming that whiteboard performance represents general aptitude rather than drill practice, some are more comfortable assuming that small assignments represent work simulation rather than plagiarism or misrepresentation. Both approaches are swamped by noise, but when you're sieving innumerable applicants and are accountable to an image of fairness and standardization, you have to use something.
It's hard to extract a signal about prospective work quality from anything but a trusted referal or a probationary hire. It's not like there's a great alternative just being ignored.
Some orgs feel more comfortable assuming that whiteboard performance represents general aptitude rather than drill practice, some are more comfortable assuming that small assignments represent work simulation rather than plagiarism or misrepresentation. Both approaches are swamped by noise, but when you're sieving innumerable applicants and are accountable to an image of fairness and standardization, you have to use something.
So if you do a take home, I believe you MUST follow it up with a live review. It becomes painfully obvious who cheated based on their ability to reason about their decisions.
In my experience, the single best question for screening out overt charlatans is a simple "Tell me about a problem you've worked on recently." Everybody engaged with technical work has a few war stories, and they're hard to fabricate without a deep understanding of the domain (at which point they wouldn't be a charlatan, now would they). It generally relaxes nervous candidates a bit, since it's external enough to not feel like bragging, while being self-congratulatory in that candidates almost always talk about a problem they succeeded in solving (and the ones who don't are always talking about a very interesting problem). What kind of problem they describe tends to give you some insight into their strengths, and their general enthusiasm clues you into their interests. Throw in some probing questions if they don't go into enough detail, and you can come out the other end of it pretty confident whether they have technical chops or not.
So far, it's been extremely successful for me in predicting whether a candidates going to get weeded out by the rest of the interviewing process or not.
So far, it's been extremely successful for me in predicting whether a candidates going to get weeded out by the rest of the interviewing process or not.
> Nothing stops someone spending 6 hours on a "2 hour" take home test
I once got rejected from a YC company because I responded more than three hours after a two hour take home test was given.
So you certainly could stop them. Doesn't really leave the best impression of the company though.
And if you're curious I wasn't actually working on it for three hours, I just started late :)
I once got rejected from a YC company because I responded more than three hours after a two hour take home test was given.
So you certainly could stop them. Doesn't really leave the best impression of the company though.
And if you're curious I wasn't actually working on it for three hours, I just started late :)
This selects for people that are chained to their email and slavishly do whatever the company says immediately -- probably intentional, and you dodged a bullet.
If a company sends me a take-home, they aren't getting it back any less than 24 hours later. I prefer if I can open it, read it and then come back to it later.
I hate it when I have to immediately open it and finish it, like a timed coding test, although I don't bother with those because I don't think I've ever passed one...
I hate it when I have to immediately open it and finish it, like a timed coding test, although I don't bother with those because I don't think I've ever passed one...
This. Email is async. I don't have notifications set up for email. I go over private email once a day. Yes even when interviewing this would only be something like once every few hours.
We can set up a time at which you will send me the take home in advance of course. Like "we will send this to you Friday around 6p.m." coz I told you that this is when I will have time for it.
You send it to me 1:37pm on a Wednesday? Forget it. I am working during that time so I won't see your email and even if I did I have no time for it. I'm working.
In the past I've even written a quick batch file to change all of the creation and modification time stamps for all files to the exact same time where they wanted a zip file. As well as setting the date and time of the git commit to a time before they sent me the take home where they wanted me to commit something to a repo.
It either goes completely unnoticed and that's fine or it makes for a nice technical convo. Meaning if someone notices and talks to me about it it would make it more likely for me to accept an offer because they have people working there already that know and understand such technical details and find them fun too. Vs. most where they would be in shock and awe how such a thing is even possible.
We can set up a time at which you will send me the take home in advance of course. Like "we will send this to you Friday around 6p.m." coz I told you that this is when I will have time for it.
You send it to me 1:37pm on a Wednesday? Forget it. I am working during that time so I won't see your email and even if I did I have no time for it. I'm working.
In the past I've even written a quick batch file to change all of the creation and modification time stamps for all files to the exact same time where they wanted a zip file. As well as setting the date and time of the git commit to a time before they sent me the take home where they wanted me to commit something to a repo.
It either goes completely unnoticed and that's fine or it makes for a nice technical convo. Meaning if someone notices and talks to me about it it would make it more likely for me to accept an offer because they have people working there already that know and understand such technical details and find them fun too. Vs. most where they would be in shock and awe how such a thing is even possible.
It's always a good idea to follow up with them in an interview, where you can ask the candidate questions about their implementation. Makes cheating much harder.
this is exactly why i reject all take homes. There is no way to compete against cheaters.
Those four companies he cites (Matter-App, Adcellerant, Symplicity, Bidsight) that tell you to spend x hours, then tell you "hey, but y'know, if you want to do [an open-ended number of additional hours for free], that'd be great!" should all go sit in the corner.
Realistically, people are going to spend more time on their take-home assignments, because they perceive there's an advantage to doing so. It's not hard to imagine how this spirals out of control over time, a feedback loop of candidates trying to do more and more free, meaningless work as the bar is raised. A three-hour assignment that takes an entire weekend, a three-hour assignment that takes a week, etc. But they shouldn't be openly encouraging this death spiral, they should be trying to keep it under control.
Better to say "if it looks like you spent 20 hours on a 3 hour assignment, we'll think of you as someone who isn't good at estimating, and wastes a lot of time fiddling with things that are out of scope".
Realistically, people are going to spend more time on their take-home assignments, because they perceive there's an advantage to doing so. It's not hard to imagine how this spirals out of control over time, a feedback loop of candidates trying to do more and more free, meaningless work as the bar is raised. A three-hour assignment that takes an entire weekend, a three-hour assignment that takes a week, etc. But they shouldn't be openly encouraging this death spiral, they should be trying to keep it under control.
Better to say "if it looks like you spent 20 hours on a 3 hour assignment, we'll think of you as someone who isn't good at estimating, and wastes a lot of time fiddling with things that are out of scope".
This is what I don't care for, because it doesn't mirror the job at all. If it takes someone 20 hours to do a 3 hour project, that's a problem and it won't show up in the takehome.
It's almost the opposite of what the job actually entails. I rarely estimate that a project will take 3 hours and then I have 48 uninterrupted hours in which to accomplish it, no biggie if it takes more than 3. The job is typically the opposite, where I estimate it will take 3 hours and then I realized that I need to accomplish 40 hours of work in the 20 meeting-free work hours I have this week so I need to find a way to finish it in 90 minutes.
It's almost the opposite of what the job actually entails. I rarely estimate that a project will take 3 hours and then I have 48 uninterrupted hours in which to accomplish it, no biggie if it takes more than 3. The job is typically the opposite, where I estimate it will take 3 hours and then I realized that I need to accomplish 40 hours of work in the 20 meeting-free work hours I have this week so I need to find a way to finish it in 90 minutes.
After reaching my 10 yoe mark, I almost explicitly refuse take home tests. Yes if there was only one company I wanted to work with and I was interviewing with only them, I'll make an exception. Companies interview a lot of candidates to hire someone, as a result of which, you as a candidate have to interview 5-10 companies to get hired and get paid fairly, 15+ if you want to maximize your career + comp. There is no reason I'll spend 40-80 hours on just take home tests alone, considering the fact that the company will still interview you for 4-5 more hours after you satisfactorily finish your take home.
If you spend 40 hours on this and it only gains you a 1% increase in salary then you earn that back in 2 years. I would say that it will generally earn you much more so personally I see it as a good investment. If you enjoy the take home you're also more likely to enjoy the role so it's also a filter for you. If you're not enjoying the take home then stop, but if you're finding it interesting then it's well worth it. These things are also for you to assess future Jon opportunities. The best take homes are as close to actual work as possible for a reason.
>I’m 27 and single. There lies the biggest advantage of a takehome assessment for me: I get to spend more time on it than someone with a wife and two kids.
I noticed the same thing. It seems take homes are mainly geared towards pre-selecting young single people with a lot of free time to code on the side besides their main job and other responsibilities, or who have no other responsibilities.
Now I'm also single and I have the time to invest in them, but I question the future of my career in tech, how will I be able to compete for new jobs later when I won't be, and take homes will be normalized? I don't remember my friends in other careers ever having to do unpaid work before getting a job. Perhaps I chose the wrong career.
What sucks even more is when you sink many hours in them and then just get ghosted, not even a rejection message, nada. I've already been burned twice by this. Or when the recruiter just spams you the take home link without first having a talk with you about the job details, the compensation range, if you're a good fit, etc. It's "first you solve this take home for us, then we can have a talk". Feels incredibly cold and inhumane.
I noticed the same thing. It seems take homes are mainly geared towards pre-selecting young single people with a lot of free time to code on the side besides their main job and other responsibilities, or who have no other responsibilities.
Now I'm also single and I have the time to invest in them, but I question the future of my career in tech, how will I be able to compete for new jobs later when I won't be, and take homes will be normalized? I don't remember my friends in other careers ever having to do unpaid work before getting a job. Perhaps I chose the wrong career.
What sucks even more is when you sink many hours in them and then just get ghosted, not even a rejection message, nada. I've already been burned twice by this. Or when the recruiter just spams you the take home link without first having a talk with you about the job details, the compensation range, if you're a good fit, etc. It's "first you solve this take home for us, then we can have a talk". Feels incredibly cold and inhumane.
> It seems take homes are mainly geared towards pre-selecting young single people with a lot of free time to code on the side besides their main job and other responsibilities, or who have no other responsibilities. (snip) I question the future of my career in tech, how will I be able to compete for new jobs later when I won't be, and take homes will be normalized?
My counterpoint to this is that takehome tests are great for folks who don't necessarily do well on brainteasers, but are otherwise strong functional developers. Despite having a family and children, I always prefer a takehome assessment. (And, just as an FYI, I'm 36 and a single parent to a young grade-schooler)
I might be biased, in that I feel like I do poorly in the live programming / brainteaser style interviews, but do strongly when I have a takehome assessment. Live programming is like a completely bizzaro-land version of programming, where a real person is staring you down while you type, observing your every interaction, and deciding your fate based on a random 60 minutes of the highest-stress part of your day. ("Oh, you googled a for loop syntax, you are clearly an idiot lying about your experience" when in fact, it's more like, "I get thrown 6 different programming languages every single week, each with similar but slightly varied syntax for every single thing, and having a person stare at me and decide the entire future of my career is a little bit stressful and anxiety-inducing").
With a takehome assessment, I can think about it for a while, I can write it all upfront, I can try a few different implementations and pick the one that feels best, I can accurately and intelligently explain exactly what I did and why I did it, I can talk in detail about the problem and potential tradeoffs. I can wrap the whole thing in nice automated testing, I can setup basic CI/CD for it on GitHub Actions or equivalent. I can demo my commitment to strong well-written clear documentation and dev experience, right in my branch Pull Request -- and all of that will likely be a closer match to what real-world day-to-day work at the new company might be like, than solving your LeetCode brainteaser.
I don't love "unpaid work before getting a job", yes that part sucks. But it sucks less than having a random stranger decide your an idiot, based on watching you sweat for an hour. Even if I'm going to get ultimately rejected anyway, the takehome route still feels better.
My counterpoint to this is that takehome tests are great for folks who don't necessarily do well on brainteasers, but are otherwise strong functional developers. Despite having a family and children, I always prefer a takehome assessment. (And, just as an FYI, I'm 36 and a single parent to a young grade-schooler)
I might be biased, in that I feel like I do poorly in the live programming / brainteaser style interviews, but do strongly when I have a takehome assessment. Live programming is like a completely bizzaro-land version of programming, where a real person is staring you down while you type, observing your every interaction, and deciding your fate based on a random 60 minutes of the highest-stress part of your day. ("Oh, you googled a for loop syntax, you are clearly an idiot lying about your experience" when in fact, it's more like, "I get thrown 6 different programming languages every single week, each with similar but slightly varied syntax for every single thing, and having a person stare at me and decide the entire future of my career is a little bit stressful and anxiety-inducing").
With a takehome assessment, I can think about it for a while, I can write it all upfront, I can try a few different implementations and pick the one that feels best, I can accurately and intelligently explain exactly what I did and why I did it, I can talk in detail about the problem and potential tradeoffs. I can wrap the whole thing in nice automated testing, I can setup basic CI/CD for it on GitHub Actions or equivalent. I can demo my commitment to strong well-written clear documentation and dev experience, right in my branch Pull Request -- and all of that will likely be a closer match to what real-world day-to-day work at the new company might be like, than solving your LeetCode brainteaser.
I don't love "unpaid work before getting a job", yes that part sucks. But it sucks less than having a random stranger decide your an idiot, based on watching you sweat for an hour. Even if I'm going to get ultimately rejected anyway, the takehome route still feels better.
How’s your career going?
I’m often worried that I’ll look like a dummy when I have to look up basic loop syntax. I’m not dumb, and I’m somewhat experienced… I just write code in Python, Matlab, Fortran, Bash, whatever else. It is shocking that the basic shape of syntax can be so different!
And also the degree to which it doesn’t actually matter. It’s just when starting a new file. My brain needs a little bit of syntax around to get locked into the language I think.
I’m often worried that I’ll look like a dummy when I have to look up basic loop syntax. I’m not dumb, and I’m somewhat experienced… I just write code in Python, Matlab, Fortran, Bash, whatever else. It is shocking that the basic shape of syntax can be so different!
And also the degree to which it doesn’t actually matter. It’s just when starting a new file. My brain needs a little bit of syntax around to get locked into the language I think.
I have been paid to write code for over 20 years, in eight or nine languages, and wouldn’t bet a serious amount of money I could fizz-buzz in any of them, including one I wrote a bunch of today, without a reference. Not without getting something basic about the language wrong in a way that would keep it from working on the first try.
This is, if anything, becoming more true the longer I do this sort of work.
This is, if anything, becoming more true the longer I do this sort of work.
My brain refuses to learn bash conditional syntax. Personally, when I’m about to write a conditional in bash I take it as a cue to switch to another language.
for i in $( ls | grep “a complicated enough filter that I never have to use conditionals in my loop”) ;
do
echo -n “if I don’t want to work on ” $i
echo “ it is time to change languages”
doneI guess an option is going into management, no take homes, DS&A groking, etc.
For me personally I prefer it to live coding as I can actually think because there's no stress but usually the "1-2h task" in reality is 8h if you want it done properly (good code, documentation, tests, project & build setup, etc.). If the job isn't top tier it is simply not worth it, at least in the previous market.
Still beats memorizing the optimal algorithms to LC questions.
For me personally I prefer it to live coding as I can actually think because there's no stress but usually the "1-2h task" in reality is 8h if you want it done properly (good code, documentation, tests, project & build setup, etc.). If the job isn't top tier it is simply not worth it, at least in the previous market.
Still beats memorizing the optimal algorithms to LC questions.
[deleted]
> mainly geared towards pre-selecting young single people with a lot of free time
Which tells you exactly how exploitive they will be with that free time. You'll onboard, it'll be good for a bit, then suddenly there will be surprise crunch-mode that will never end.
> when you sink many hours in them and then just get ghosted, not even a rejection message
This will manifest in a thank you of a layoff after all those crunch hours.
> first you solve this take home for us, then we can have a talk
"Cool, I can take on that contract. I'll put together a quote for you and I'll get started. Who should I send my W-9 to?" Many will, of course, balk at this, which tells you everything you need to know about them. The good ones will pay it. Set your boundaries.
Which tells you exactly how exploitive they will be with that free time. You'll onboard, it'll be good for a bit, then suddenly there will be surprise crunch-mode that will never end.
> when you sink many hours in them and then just get ghosted, not even a rejection message
This will manifest in a thank you of a layoff after all those crunch hours.
> first you solve this take home for us, then we can have a talk
"Cool, I can take on that contract. I'll put together a quote for you and I'll get started. Who should I send my W-9 to?" Many will, of course, balk at this, which tells you everything you need to know about them. The good ones will pay it. Set your boundaries.
I think this is a valid use case. My concern with thing like that is that it can be abused relatively easily. Long time ago when I was a little boy, in the old country translating to/from English to earn some additional money while studying, I came across the practice of using students ( giving them a portion of the assignment and just run the project through applicants ). Naturally, if you are just starting, in tech it may be easier to judge if it is a real test of skill.
In my engineering days, I'd agree with this. IF we have spent a good amount of time talking and IF there is an affirmation of "we WILL review this code with you" (not just "if we like what we see, then we'll talk"), then I'd -consider- it.
If your idea of hiring is to spam me after I apply in Workday with a multi hour coding project to see if I merit the time speaking to even a recruiter, no, that job app is going in my circular file.
If your idea of hiring is to spam me after I apply in Workday with a multi hour coding project to see if I merit the time speaking to even a recruiter, no, that job app is going in my circular file.
Having interviewed hundreds of consultants and been interviewed by many and having seen the cheating etc. from all sides I have first hand opinion on this.
Sadly how people recruit is completely broken for most companies that are not FAANG etc. (It is broken for FAANG as well but in a different way).
As I often say "you need to win, not win an argument". Similarly you want candidates that can be good productive members of your team at the salary you are paying and not the candidates who can nail your interview process.
The problem with take home tests is not the cheating. But rather failure of the recruiter to actually evaluate the assignment. Do you care about the end result or do you care about the art of craft ? Do you care about readability of the code or performance of the app ? Is the assignment similar to the work candidate will end up doing at workplace ?
My suggestion generally has been:
1. Look at candidate's past work. Has candidate worked with reputed companies ? 2. Talk with candidate with general technology topics around their work. Something like "what do you like about react?" 3. Give candidate a very simple codesignal test. Nothing to fancy. Say if you want react engineers just test their ability to implement simple components.
This vastly kills a lot of inferior talent in my opinion.
4. Give a very simple problem without any "trick" or "iq test" and ask the candidate to code it in front of you. Also let the candidate use internet, documentation and AI tools.
In this day and age asking candidates NOt to use AI tools like asking a candidate to write code without using keyboard. You want people who can use things like Github copilot.
Sadly how people recruit is completely broken for most companies that are not FAANG etc. (It is broken for FAANG as well but in a different way).
As I often say "you need to win, not win an argument". Similarly you want candidates that can be good productive members of your team at the salary you are paying and not the candidates who can nail your interview process.
The problem with take home tests is not the cheating. But rather failure of the recruiter to actually evaluate the assignment. Do you care about the end result or do you care about the art of craft ? Do you care about readability of the code or performance of the app ? Is the assignment similar to the work candidate will end up doing at workplace ?
My suggestion generally has been:
1. Look at candidate's past work. Has candidate worked with reputed companies ? 2. Talk with candidate with general technology topics around their work. Something like "what do you like about react?" 3. Give candidate a very simple codesignal test. Nothing to fancy. Say if you want react engineers just test their ability to implement simple components.
This vastly kills a lot of inferior talent in my opinion.
4. Give a very simple problem without any "trick" or "iq test" and ask the candidate to code it in front of you. Also let the candidate use internet, documentation and AI tools.
In this day and age asking candidates NOt to use AI tools like asking a candidate to write code without using keyboard. You want people who can use things like Github copilot.
I don't know about your first point. We recently hired two persons who came from VeryBigBusiness Inc. and VeryImportantCorp Inc., so I assumed that they were at least slightly above the average. Soon it became evident that the only field were they both excel was the art of arranging unending meetings, being very vocal in those meetings, looking always too stressed and too busy, getting themselves in key projects while avoiding the day to day work, and never getting any real work done.
One of them was causing such friction with endless meetings and mail chains, that a project she coordinated that was a quagmire for six months was finished in the two weeks after she left.
One of them was causing such friction with endless meetings and mail chains, that a project she coordinated that was a quagmire for six months was finished in the two weeks after she left.
It is at the point where for me when they ask for a take home, it is a self selecting: Thanks but no thanks.
I can spend time answering your take-home, or trying to get other interviews. From what I've seen on take homes: Other interviews are strongly EV+.
Before you go: But you never got a job off a take home, that's why. I did. It was one of the worst managed misfits I've had. I am sure others thrived there, for me it was a living hell, that didn't even pay that great.
So... Much like requiring a suit to interview and a few other things. It goes in my bin of "Your company has performed self selection."
Note: I will leetcode, at least there is a human there, we're talking, interacting, and I am learning about the company.. But most of all, that feel of: I put up and hour, you put up an hour. Let's talk. Is very real.
I can spend time answering your take-home, or trying to get other interviews. From what I've seen on take homes: Other interviews are strongly EV+.
Before you go: But you never got a job off a take home, that's why. I did. It was one of the worst managed misfits I've had. I am sure others thrived there, for me it was a living hell, that didn't even pay that great.
So... Much like requiring a suit to interview and a few other things. It goes in my bin of "Your company has performed self selection."
Note: I will leetcode, at least there is a human there, we're talking, interacting, and I am learning about the company.. But most of all, that feel of: I put up and hour, you put up an hour. Let's talk. Is very real.
I think the timer is probably something that doesn't get enough attention here. Lots of people say about how these select for people with lots of free time, and that's very true, but they also seem to be biased towards people that don't tend to freak out in stressful situations/do well with a clock rapidly ticking down behind them.
Which... doesn't necessarily correlate to someone being good at programming or web development or what not. Or whether they'll be good at the job in general, given that most engineering roles don't have someone screaming "get this done in under an hour or else!" with no time to think things through properly.
At the same time though, I can also see why they do that, since otherwise it comes down to "whoever's got the most free time and willingness to grind away at the problem", turning an hour long exercise into a 3 day one in the process.
Which... doesn't necessarily correlate to someone being good at programming or web development or what not. Or whether they'll be good at the job in general, given that most engineering roles don't have someone screaming "get this done in under an hour or else!" with no time to think things through properly.
At the same time though, I can also see why they do that, since otherwise it comes down to "whoever's got the most free time and willingness to grind away at the problem", turning an hour long exercise into a 3 day one in the process.
Founder of new tech assessment company / mentioned in article here.
We're biased, but we think the old form of take-home assessments (+ classic Leetcode tests, etc) are completely broken. Beyond reasons you all mention - they're completely unreliable today in the age of ChatGPT, etc. Way too easy to cheat.
We're seeing candidates copy takehome instructions into an LLM, paste the solution and submit in <5 minutes. It's hard to write a problem that's (1) low enough scope to solve in a short time, but (2) hard enough that LLMs can't step towards sovling it.
At Ropes - we're using LLMs to evaluate how candidates code, not just the final solution. So HM's can step in and look at a timeline of actions taken to reach the solution, instead of just the final answer. How do candidates debug? What edge cases do they consider? Etc. We think these answers hold real signal and can be answered for the first time async.
We're trying to make this better for candidates too. E.g. (1) shorter assessments, (2) you can often use your own IDE, (3) you're not purely evaluated on test cases, etc. But we're not yet perfect. If this sounds interesting / you have strong thoughts I'd love to talk to you - email is in my bio.
We're biased, but we think the old form of take-home assessments (+ classic Leetcode tests, etc) are completely broken. Beyond reasons you all mention - they're completely unreliable today in the age of ChatGPT, etc. Way too easy to cheat.
We're seeing candidates copy takehome instructions into an LLM, paste the solution and submit in <5 minutes. It's hard to write a problem that's (1) low enough scope to solve in a short time, but (2) hard enough that LLMs can't step towards sovling it.
At Ropes - we're using LLMs to evaluate how candidates code, not just the final solution. So HM's can step in and look at a timeline of actions taken to reach the solution, instead of just the final answer. How do candidates debug? What edge cases do they consider? Etc. We think these answers hold real signal and can be answered for the first time async.
We're trying to make this better for candidates too. E.g. (1) shorter assessments, (2) you can often use your own IDE, (3) you're not purely evaluated on test cases, etc. But we're not yet perfect. If this sounds interesting / you have strong thoughts I'd love to talk to you - email is in my bio.
> The majority of the companies I noticed issuing takehomes weren’t your star companies. They weren’t FAANG. They weren’t looking to change the world. They weren’t looking for olympiad champions or kumon alumnis.
> Rather, they were the design studios, the lean web development agencies, the mobile dev teams based out of Eastern Europe.
Given a team of... let's make up some numbers.... {8} devs working on client projects that have {250} candidates to evaluate, how much time do the {senior} members of the team have to evaluate them? Note that taking {two} senior devs in such a shop out for {30 minutes} for each candidate for a live coding interview is {=about a week and a half} of nothing but doing interviews for each of them - removing them from other revenue generating tasks and a reduction of {at least 1/4th} of the work for that duration.
Extending this beyond 2 weeks of time means that the best candidates are likely to have found other opportunities.
Take home tasks shifts that time to a standardized "I can look at this sample code in 2-3 minutes and get a feel of if the candidate will be producing something maintainable or hot garbage."
At that point, one can narrow down the pool of candidates down to a small number that can be interviewed and decided with some expedience.
> Rather, they were the design studios, the lean web development agencies, the mobile dev teams based out of Eastern Europe.
Given a team of... let's make up some numbers.... {8} devs working on client projects that have {250} candidates to evaluate, how much time do the {senior} members of the team have to evaluate them? Note that taking {two} senior devs in such a shop out for {30 minutes} for each candidate for a live coding interview is {=about a week and a half} of nothing but doing interviews for each of them - removing them from other revenue generating tasks and a reduction of {at least 1/4th} of the work for that duration.
Extending this beyond 2 weeks of time means that the best candidates are likely to have found other opportunities.
Take home tasks shifts that time to a standardized "I can look at this sample code in 2-3 minutes and get a feel of if the candidate will be producing something maintainable or hot garbage."
At that point, one can narrow down the pool of candidates down to a small number that can be interviewed and decided with some expedience.
The flip side is that there’s no way I’m doing a “two hour” (really 4-8 hour) take-home if I’m still at the stage where there are 250 candidates. Or 25, even. Maybe three.
The next question would be "how do you narrow from {250} down to {25} by spending at most 3 minutes per resume?"
The answer there is often "select only the ones that show the most things on the resume without evaluation of their practical skills" ... and we get to the complaints then of "but people lie on their resumes all the time" and "but my skills at coding aren't things that I can reflect on the resume?"
And when you're spending 3-4 minutes per resume, you don't have time to open up a GitHub link and try to figure out if the person who this is wrote the code or if they just forked another repo ... and what contributions are theirs.
For evaluating a coder, the time imbalance is heavy on one side or the other.
Having two people do seven hours of interviews a day for two weeks is also in the "not feasible" category. As its the company that is setting up the criteria for the interview, they've got the first move and are setting up a process that they can fairly evaluate the candidates that apply and go forward with that step in a way that is most likely to eliminate the most risky candidates and find the candidates that have the skills but don't present well on a resume.
The answer there is often "select only the ones that show the most things on the resume without evaluation of their practical skills" ... and we get to the complaints then of "but people lie on their resumes all the time" and "but my skills at coding aren't things that I can reflect on the resume?"
And when you're spending 3-4 minutes per resume, you don't have time to open up a GitHub link and try to figure out if the person who this is wrote the code or if they just forked another repo ... and what contributions are theirs.
For evaluating a coder, the time imbalance is heavy on one side or the other.
Having two people do seven hours of interviews a day for two weeks is also in the "not feasible" category. As its the company that is setting up the criteria for the interview, they've got the first move and are setting up a process that they can fairly evaluate the candidates that apply and go forward with that step in a way that is most likely to eliminate the most risky candidates and find the candidates that have the skills but don't present well on a resume.
That’s offloading nearly a quarter of a person-year of work onto the 250 candidates—potentially much more, if it’s more than a truly-two-hour assignment. To save yourselves a far smaller amount of work. And that’s for just one stage of the interview process!
249 will receive no benefit for their work.
Nobody with options is gonna take that offer and you’ll be left with the desperate. It’s beyond rude, it’s a bad joke. If desperate’s what you want, then I guess go for it.
249 will receive no benefit for their work.
Nobody with options is gonna take that offer and you’ll be left with the desperate. It’s beyond rude, it’s a bad joke. If desperate’s what you want, then I guess go for it.
I am certainly not saying that it is time fair or that it doesn't result in a lot of excess work being done.
Take home assignments are there to minimize the amount of time spent per candidate with the most limited resource in the interview process - the people on a small team doing the interviews.
That is the problem that it is trying to solve.
Trying to minimize for the candidate time spent, increases the amount of the limited resource (the interviewers' time) and may mean that the interview process will be longer than is acceptable for candidates (are you willing to wait 4 weeks between the last interview and offer?) and removes the interviewers from revenue generating tasks.
There is no good solution to this. Often take home assignments are the least bad solution.
Take home assignments are there to minimize the amount of time spent per candidate with the most limited resource in the interview process - the people on a small team doing the interviews.
That is the problem that it is trying to solve.
Trying to minimize for the candidate time spent, increases the amount of the limited resource (the interviewers' time) and may mean that the interview process will be longer than is acceptable for candidates (are you willing to wait 4 weeks between the last interview and offer?) and removes the interviewers from revenue generating tasks.
There is no good solution to this. Often take home assignments are the least bad solution.
Would be curious to get people's input - Assuming you were a candidate with a decent resume but no significant public Github activity, would it be an interesting alternative to a take home assessment or in person white board coding session, if a hiring manager suggested that you fork an open source project of your choice and make a PR of your choice (or other contribution)?
I don't think we'll ever solve the problem of interviewing software engineers.
- Whiteboard coding tasks put a LOT of pressure on you to perform while being watched in a stressful high-stakes situation, which is a situation that the extreme majority of SWEs won't deal with, and many simply can't. They also tend to test for algorithms (inverting a binary tree, etc.), something that is unrelated to most actual jobs.
- Takehome assessments are unfair to those with a family that might not have the free time to do them, and if you're applying for multiple jobs, can easily become a full-time job on their own.
- Performing a 5+ round interview gauntlet is frustrating and absurd, and can make the hiring process take well over a month.
- Whiteboard coding tasks put a LOT of pressure on you to perform while being watched in a stressful high-stakes situation, which is a situation that the extreme majority of SWEs won't deal with, and many simply can't. They also tend to test for algorithms (inverting a binary tree, etc.), something that is unrelated to most actual jobs.
- Takehome assessments are unfair to those with a family that might not have the free time to do them, and if you're applying for multiple jobs, can easily become a full-time job on their own.
- Performing a 5+ round interview gauntlet is frustrating and absurd, and can make the hiring process take well over a month.
I feel like someone should cut a check after submitting someone to 5+ rounds. Or at least like a $200 Amazon gift card.
This:
> If the very first thing in your application process is a 6 hour long unpaid assignment, then it’s hard to imagine you’re not wasting someone’s time because at this stage you’ve barely done any filtering.
I pretty much walk away at this point. It's like asking a person for sex after buying them a drink. (And if I have to explain why, you clearly don't get it.)
> But if the takehome is offered, say, after 3 rounds of interviews and will be discussed in detail during your final round with a panel of two senior engineers, I think that’s fair and provides a strong signal.
Exactly. That's the point that I'm willing to invest in a takehome.
> If the very first thing in your application process is a 6 hour long unpaid assignment, then it’s hard to imagine you’re not wasting someone’s time because at this stage you’ve barely done any filtering.
I pretty much walk away at this point. It's like asking a person for sex after buying them a drink. (And if I have to explain why, you clearly don't get it.)
> But if the takehome is offered, say, after 3 rounds of interviews and will be discussed in detail during your final round with a panel of two senior engineers, I think that’s fair and provides a strong signal.
Exactly. That's the point that I'm willing to invest in a takehome.
It’s just astounding to me that people can pay $x00,000 to spend 4-10 years on a single discipline, and still not have the minimal required proof of competence to get a job.
And how are these big tech firms spending $billions on recruiting and interview time, as if the education credential is worthless, but still preferring to use it as a first-line filter?
And how are these big tech firms spending $billions on recruiting and interview time, as if the education credential is worthless, but still preferring to use it as a first-line filter?
I had a previous job that paid me for the time it took to do a takehome interview. (it ended up being right at the minimum before having to report it as taxable income). I highly welcome this, as it signals a commitment from the company that they're serious about the candidate
The solution is simple: pay candidates for the estimated time the assignment should take, based on the median rate of the role they're applying for. Pay them regardless if they pass or fail the test. You should be able to avoid grifters looking for handouts if your screening process is any good.
But I don't think this type of evaluation is a valuable signal anymore in the age of AI. Anyone could nowadays finish your take home assignment in a fraction of the effort it would take them without AI. This might not be relevant after they're hired, since they would likely also use AI assistants, but for evaluation purposes you want to assess how someone thinks about problem solving, not how they generate the solution, and then confidently walk you through it.
The best interview style for software developers IMO is the code review challenge. Show them a piece of code from a hypothetical PR of a junior developer, and ask them to a) describe what the code does, b) point out any issues they see, and c) fix the issues. You can have several difficulty levels depending on the role and expected candidate experience, or you can cherry pick these from your own codebase. The benefits of this approach are that it's a simulation of a real-world task they would be doing on a daily basis, it's a collaborative effort and showcases their communication skills, while also allowing them to evaluate yours, it's much less stressful than white boarding and quizzing sessions though still involves some coding, and unlike take home assignments, it can be completed in an hour or 90 minutes at most.
But I don't think this type of evaluation is a valuable signal anymore in the age of AI. Anyone could nowadays finish your take home assignment in a fraction of the effort it would take them without AI. This might not be relevant after they're hired, since they would likely also use AI assistants, but for evaluation purposes you want to assess how someone thinks about problem solving, not how they generate the solution, and then confidently walk you through it.
The best interview style for software developers IMO is the code review challenge. Show them a piece of code from a hypothetical PR of a junior developer, and ask them to a) describe what the code does, b) point out any issues they see, and c) fix the issues. You can have several difficulty levels depending on the role and expected candidate experience, or you can cherry pick these from your own codebase. The benefits of this approach are that it's a simulation of a real-world task they would be doing on a daily basis, it's a collaborative effort and showcases their communication skills, while also allowing them to evaluate yours, it's much less stressful than white boarding and quizzing sessions though still involves some coding, and unlike take home assignments, it can be completed in an hour or 90 minutes at most.
I really like the PR-role-play interview, too. I do worry that it can devolve into the classic mind-reading interview—i.e., there is an extremely subtle bug (or perceived deficiency) in the code that a candidate is dinged for not seeing.
This is exactly what I do. It's rare that someone is required to create something entirely from scratch. Most of what we do as developers is altering an existing system. I want to select for people who can figure out how to do that.
Also it completely eliminates the tool selection and test system portions of the exercise because I pick all of that stuff and make sure anything I don't care about is automated or is out of scope of the solution.
Done correctly you can usually get through it in less than an hour, leaving about 10 or 15 minutes for the candidate to ask me some questions.
Also it completely eliminates the tool selection and test system portions of the exercise because I pick all of that stuff and make sure anything I don't care about is automated or is out of scope of the solution.
Done correctly you can usually get through it in less than an hour, leaving about 10 or 15 minutes for the candidate to ask me some questions.
+1
I am totally surprised that "reading code" is not really appreciated even at FAANG companies though very likely that is one of the most important things they will end up doing.
I am totally surprised that "reading code" is not really appreciated even at FAANG companies though very likely that is one of the most important things they will end up doing.
I've yet to go through one of these, but I've at least imagined it would be a decent format.
(For me, at least. I don't parallel-task very well in the first place, and being under the magnifying glass makes it significantly worse. I have trouble, say, even making sense of the words someone else says when I'm already focused on coding/reading/writing something. If the interviewer interjects a thought or question about what I'm doing, I often have to stop, struggle to ~park my train of thought, and then ask them to repeat what they said. If I make the mistake of asking them to repeat before the train's parked, I may have to ask them to repeat it again.)
I think I'd also look pretty favorably on an interviewing process that gives a few format choices and lets me choose what I'm most comfortable with.
(For me, at least. I don't parallel-task very well in the first place, and being under the magnifying glass makes it significantly worse. I have trouble, say, even making sense of the words someone else says when I'm already focused on coding/reading/writing something. If the interviewer interjects a thought or question about what I'm doing, I often have to stop, struggle to ~park my train of thought, and then ask them to repeat what they said. If I make the mistake of asking them to repeat before the train's parked, I may have to ask them to repeat it again.)
I think I'd also look pretty favorably on an interviewing process that gives a few format choices and lets me choose what I'm most comfortable with.
we had one of these at a previous job, and I was vocally against it the whole time. Even though it did weed out a few clueless people that seemed to know what they were doing during the interview.
the project was to make a django app for storing your pokemon application, and a script to download the list of pokemon from an api. So atleast it was fun, but i hated turning people down who took the time to do it.
the project was to make a django app for storing your pokemon application, and a script to download the list of pokemon from an api. So atleast it was fun, but i hated turning people down who took the time to do it.
What I hate is that there's no often no discussion about your solution. I want to go "here, this is my solution", and then have a discussion with the people hiring where they might ask stuff like "you chose to do X this way, what was your reasoning?"
Also, enough with the fucking Roman numerals, Jesus Christ.
Also, enough with the fucking Roman numerals, Jesus Christ.
I recently designed a take-home. I made sure to pick a real, interesting, problem we faced, and one where there is a very high dynamic range of responses - it can distinguish between good and bad juniors (I know this because I let all my students try it if they want, which also gave me great deal flow of juniors without even needing to spend time on screening the bad ones), and it gives some information even at my level (I didn't arrive at the best possible solution myself, someone came up with a better solution to one subproblem).
I like respecting candidates' time, because I want the strongest candidates to not filter themselves out at the mention of a take-home. That's why I ask them to schedule 4 hours with me where at the start they will get the problem description and at the end they will submit what they have - this way they aren't pressured to put in more time because they know for sure that their competitors didn't - and I get to compare apples to apples.
I also designed the rules to minimize the perverse incentives set by the clock by giving multiple ways to get recognition for partial solutions.
This is the content of the document the candidate gets initially (they only get the actual problem later):
Meta
This is a real task we encountered and solved at Finubit. Out of respect for your time, we limit this task to 4 hours (of wallclock time, to remove the incentive to spend more than that to gain a competitive advantage). It took us much more than four hours to get the perfect solution, so don't worry, we don’t expect a full production-ready solution to 100% of the problem.
We ask that you approach this task as you would the first 4 hours out of as much time as you need to finish the task to a reasonable quality standard. We do want to see a working Proof Of Concept at the end of those 4 hours, like we would ask of any team member picking up a long, open-ended research task. The POC can be very limited in functionality, but functionality that it promises will be held to a standard of quality.
If you encounter ambiguities or mistakes, assume the most reasonable way to resolve them and make a note to check with Product that your understanding was correct. If you disagree with Product about something, make a note to push back against the design decision they made (but in the meantime assume the current spec, unless it's grossly wrong). If there's a subcomponent in your design that you don't know how to implement but believe to be solvable, it's OK to assume the existence of a black box that solves it.
Please take a moment to set an alarm clock. When the clock strikes 4 hours, please send us your code and research notes, and then go over the code and document everything you’d change before you consider the code production-ready and send us the annotated code.
We will be available to answer clarification questions if needed.
Then there is a checklist to reduce loss on technicalities (forgetting to set an alarm or send annotations).
(If you're in Israel, and you want to try it for fun, shoot me an email)
I like respecting candidates' time, because I want the strongest candidates to not filter themselves out at the mention of a take-home. That's why I ask them to schedule 4 hours with me where at the start they will get the problem description and at the end they will submit what they have - this way they aren't pressured to put in more time because they know for sure that their competitors didn't - and I get to compare apples to apples.
I also designed the rules to minimize the perverse incentives set by the clock by giving multiple ways to get recognition for partial solutions.
This is the content of the document the candidate gets initially (they only get the actual problem later):
Meta
This is a real task we encountered and solved at Finubit. Out of respect for your time, we limit this task to 4 hours (of wallclock time, to remove the incentive to spend more than that to gain a competitive advantage). It took us much more than four hours to get the perfect solution, so don't worry, we don’t expect a full production-ready solution to 100% of the problem.
We ask that you approach this task as you would the first 4 hours out of as much time as you need to finish the task to a reasonable quality standard. We do want to see a working Proof Of Concept at the end of those 4 hours, like we would ask of any team member picking up a long, open-ended research task. The POC can be very limited in functionality, but functionality that it promises will be held to a standard of quality.
If you encounter ambiguities or mistakes, assume the most reasonable way to resolve them and make a note to check with Product that your understanding was correct. If you disagree with Product about something, make a note to push back against the design decision they made (but in the meantime assume the current spec, unless it's grossly wrong). If there's a subcomponent in your design that you don't know how to implement but believe to be solvable, it's OK to assume the existence of a black box that solves it.
Please take a moment to set an alarm clock. When the clock strikes 4 hours, please send us your code and research notes, and then go over the code and document everything you’d change before you consider the code production-ready and send us the annotated code.
We will be available to answer clarification questions if needed.
Then there is a checklist to reduce loss on technicalities (forgetting to set an alarm or send annotations).
(If you're in Israel, and you want to try it for fun, shoot me an email)
In my last job search (~3 years ago) I was presented with many requests to complete a design challenge. I rejected them outright and the responses I got (typically from a 3rd party "design recruiter") were quite astounding. Some acknowledged and moved on, but there were several who clearly expressed frustration, disdain, sometimes almost anger. One dropped me from another role I was working with them on for refusing.
Now it's a total red flag for me. But judging from Blind posts it's still a common practice.