Ask HN: How to design a good technical interview test (take home or live)
What are some principles in test design that make it revealing and easy to time-box, regardless of role/stack? Any specific test you designed or solved as an interviewee that stands out?
20 comments
Good advice here: https://jacobian.org/series/work-sample-tests/ (also take a look at the series on interview questions and what makes them more or less predictive of performance)
Love this! Jacob's posts on work sample tests are the most comprehensive I've seen on the topic.
I don't think that take-at-home tests are fair. Most of times I've been given one the 3-4 hours have been actually a weekend. In fact, I have the policy of only accepting take-at-home tests that are interesting problems that help me learn something along the process of developing them.
On the other hand, I also don't like on-site interviews. A random question not-related at all with the position you're interviewing for is given, and have to solve it while you're beholded by a bunch of strangers... Being tired/nervous after a 3/4 hour in-site interview, scenic panic... To sum up, tThe recipe for disaster.
Whe I was the interviewer long ago, I gave them a questionary that could be solved in 1 hour. I made sure to tell the candidates that don't get stressed, look for whatever info you want in internet, and that there's no good/bad answers. In fact, I gave them the questionary and once the time has passed I would get them to check the answers.
There was several level of questions, to prune candidates at different levels. That also helped to determine the level of the candidate (I was working for a 200 employee non-tech company at the time).
I suppose that the questionary could have been asked to be completed in-place, in an empty room and without internet... But, what's the point of that? In the real world a developer is allowed to use internet, in fact, knowing to search is a skill I'd like to check.
Besides, if they had some open-source projects I would take a look at that, and at the end, there was a technical conversation between them and some senior developers about their career.
On the other hand, I also don't like on-site interviews. A random question not-related at all with the position you're interviewing for is given, and have to solve it while you're beholded by a bunch of strangers... Being tired/nervous after a 3/4 hour in-site interview, scenic panic... To sum up, tThe recipe for disaster.
Whe I was the interviewer long ago, I gave them a questionary that could be solved in 1 hour. I made sure to tell the candidates that don't get stressed, look for whatever info you want in internet, and that there's no good/bad answers. In fact, I gave them the questionary and once the time has passed I would get them to check the answers.
There was several level of questions, to prune candidates at different levels. That also helped to determine the level of the candidate (I was working for a 200 employee non-tech company at the time).
I suppose that the questionary could have been asked to be completed in-place, in an empty room and without internet... But, what's the point of that? In the real world a developer is allowed to use internet, in fact, knowing to search is a skill I'd like to check.
Besides, if they had some open-source projects I would take a look at that, and at the end, there was a technical conversation between them and some senior developers about their career.
I don't feel that take-home tests are fair because of the time that is required, but on the other side of it, I excel at them and have gotten my best jobs from returning excellent results on take-home projects in a short amount of time. For some reason, the format just works better for me and it takes a lot of the pressure out of the process. The downsides become more apparent when you're interviewing for several jobs at the same time and have more than one take-home assignment.
I think lots of companies compensate for take-home tests (as we are doing now). Also I think the willingness to do a take-home is in itself a good filter (but you gotta make it interesting, as parent said)
That's right. That's the main point in favor of making the candidate do a simple questionnaire in office (or remote but in real time). Only 1 hour of their time is wasted.
I think you mean that poorly designed take-home tests aren't fair. Do you think a well-designed, hourlong one could be fair?
I'm genuely interested, what kind of test could that be? Some tests they have gave me in several interviews:
- Online shops.
- Hackerrank questions.
- Read from a big log file in an efficient manner with Python (did it but there was also a 3-4 hour on-site interview).Any test that:
1) is similar to the work in the day-to-day role
2) doesn't require more than a 1-2 hour time commitment from a candidate
3) is interesting
HackerRank questions typically don't meet criteria 1 or 3. The best tests are often ones that the engineering team has designed themselves.
For example, this is an example of a test that meets all the criteria above: https://app.mightyacorn.io/flow-club/founding-front-end-engi...
HackerRank questions typically don't meet criteria 1 or 3. The best tests are often ones that the engineering team has designed themselves.
For example, this is an example of a test that meets all the criteria above: https://app.mightyacorn.io/flow-club/founding-front-end-engi...
That sounds like you want to keep people away from your organization. Why not ask them these questions during the interview? Usually ask questions around the issues that your organization is facing. See how they would solve that problem. Often times, teams get intimidated by strong talent so they often pass on these scouts, but your team should be the one that is providing you with the problems for these candidates to solve their riddles =p
Always present real time challenges, so that you are helping the teams pass obstacles and not providing tests that make no sense in teams therefore resulting in steep learning curves.
Always present real time challenges, so that you are helping the teams pass obstacles and not providing tests that make no sense in teams therefore resulting in steep learning curves.
None of this at all. People already take time for multiple interview stages. Ask targeted questions, but don't trouble them with more 'work'. Ask for references.
The principles below are specifically for take-home tests, though some will still apply to technical interviews conducted live:
1. Design tests to be open-ended and demonstrate how a candidate thinks, rather than ones that drive toward a binary pass/fail outcome. Asking why a candidate worked on the pieces they did and where they would focus if given more time gets you high signal without requiring as much time from the candidate.
2. Provide clarity on the grading rubric. When candidates understand how they'll be evaluated, they'll spend less time on work that doesn't result in higher signal to the company. For example, having a candidate write some tests or talk through how they would approach testing is great! Expecting full test coverage isn't as useful.
3. Scoping is the toughest part about test design. Try to choose topics that closely mirror the day-to-day work in the role itself but that can be boiled down to a 1-2 hour task. You want to make sure that candidates who perform well are more likely to succeed if hired, but you also want to keep it short. From interviewing 50 hiring managers who designed the hiring processes at Basecamp/Medium/Mailchimp/etc, they revealed that the candidates who were ultimately hired consistently spent 2x the recommended time, which means that designing a 3-hour take-home is asking top candidates to commit 6 hours. It's a tough ask in the current hiring climate.
Here are 3 examples for inspiration:
- Full-stack role for Flow Club (YC S21) that runs virtual WeWork sessions: https://app.mightyacorn.io/flow-club/founding-front-end-engi...
- Front-end role for Playhouse (YC S21), building TikTok for real estate: https://app.mightyacorn.io/demo/playhouse-hiring-challenge
- Back-end role for zero5, a parking automation startup: https://app.mightyacorn.io/zero5/challenge-fullstack-softwar...
Happy to share more details if you have specific questions!
Source: my team spent the past 6 months designing personalized take-home tests for 20 VC-backed startups hiring founding engineers. We evaluated ~500 candidate submissions.
1. Design tests to be open-ended and demonstrate how a candidate thinks, rather than ones that drive toward a binary pass/fail outcome. Asking why a candidate worked on the pieces they did and where they would focus if given more time gets you high signal without requiring as much time from the candidate.
2. Provide clarity on the grading rubric. When candidates understand how they'll be evaluated, they'll spend less time on work that doesn't result in higher signal to the company. For example, having a candidate write some tests or talk through how they would approach testing is great! Expecting full test coverage isn't as useful.
3. Scoping is the toughest part about test design. Try to choose topics that closely mirror the day-to-day work in the role itself but that can be boiled down to a 1-2 hour task. You want to make sure that candidates who perform well are more likely to succeed if hired, but you also want to keep it short. From interviewing 50 hiring managers who designed the hiring processes at Basecamp/Medium/Mailchimp/etc, they revealed that the candidates who were ultimately hired consistently spent 2x the recommended time, which means that designing a 3-hour take-home is asking top candidates to commit 6 hours. It's a tough ask in the current hiring climate.
Here are 3 examples for inspiration:
- Full-stack role for Flow Club (YC S21) that runs virtual WeWork sessions: https://app.mightyacorn.io/flow-club/founding-front-end-engi...
- Front-end role for Playhouse (YC S21), building TikTok for real estate: https://app.mightyacorn.io/demo/playhouse-hiring-challenge
- Back-end role for zero5, a parking automation startup: https://app.mightyacorn.io/zero5/challenge-fullstack-softwar...
Happy to share more details if you have specific questions!
Source: my team spent the past 6 months designing personalized take-home tests for 20 VC-backed startups hiring founding engineers. We evaluated ~500 candidate submissions.
These are pretty awesome looking take-home interviews! I run a job board for companies that don’t do LeetCode interviews and would be interested in getting these companies and others added in. May end up emailing you to see if I can get more details!
Sounds similar to Hiring Without Whiteboards. I'd love to learn more - you can reach me at [email protected]
1. Focus on a simplified version of a real problem you had to solve at your current job (MIN(arbitrary))
2. Needs to be explainable in 1 page with example data (MIN(completely))
3. Needs to not require any complex interaction. No web request, no complex libraries, just a unit testable piece of code. (MAX(impl speed))
4. Make it fun and something you and all your coworkers could easily do.
This has been my approach and it's been a good filter. If it's done before the interview have them come in and add a feature. Make sure they bring a machine with their prefered IDE / editor to maximize similarity to their real programming flow. Offer to do something else while they code so they don't feel "watched"
2. Needs to be explainable in 1 page with example data (MIN(completely))
3. Needs to not require any complex interaction. No web request, no complex libraries, just a unit testable piece of code. (MAX(impl speed))
4. Make it fun and something you and all your coworkers could easily do.
This has been my approach and it's been a good filter. If it's done before the interview have them come in and add a feature. Make sure they bring a machine with their prefered IDE / editor to maximize similarity to their real programming flow. Offer to do something else while they code so they don't feel "watched"
I don't like take homes, because you are able to cheat... (ask a friend, hire a fiverr, etc.) - so a brilliant solution is not very helpful to find out, if you really met an expert.
I prefer a 2 - 6 hour training day (depending on whats possible) for the best applicant, where he / she meets the team, sees how they are working and most of the people I met, were interested, because they benefit from this, too.
When you MUST provide a take home one, you could take a look at adventofcode problems as inspiration. They are fun to do, not too complicated and do not depend on a specific technology.
I prefer a 2 - 6 hour training day (depending on whats possible) for the best applicant, where he / she meets the team, sees how they are working and most of the people I met, were interested, because they benefit from this, too.
When you MUST provide a take home one, you could take a look at adventofcode problems as inspiration. They are fun to do, not too complicated and do not depend on a specific technology.
I encourage you to see what happens when you leave testing out of the interview.
Learn about them, ask them to participate in a code review if you have to, pretend you work with them already.
Learn about them, ask them to participate in a code review if you have to, pretend you work with them already.
Don't. My best interview where ones where we talk about my open source contributions. One would be enough, but I have a few because I can work on my employer own open source project.
For jobs that I am excited about I wouldn't mind doing a take home test as long as the project was interesting.
no take homes pls