Guide to Take-home Coding Challenges(fullstackinterviewing.com)
fullstackinterviewing.com
Guide to Take-home Coding Challenges
https://www.fullstackinterviewing.com/2018/02/02/the-ultimate-guide-to-kicking-ass-on-take-home-coding-challenges.html
149 comments
This is my gripe with take homes. I have spent a week on a take home before and got glowing reviews on how awesome the tests were etc. only to be rejected down the line without any feedback whatsoever. If I apply for a job now and they give me a takehome I kindly ask for an in person interview or decline to proceed with the process.
This is tough - I've had similar experiences. I think the main issue here is that the company didn't give you any feedback, so it's hard to know what happened. I've submitted a take-home challenge, but had hiring needs change in the time it took me to complete it, so the company passed. Though I think there are a ton of things you can do to make your self stand out as a candidate with a take-home challenge vs. a whiteboarding interview, they are not for everyone.
I'm pretty sure no feedback is the norm, not the exception. I've done two coding challenges. One was reasonable and lead to an interview, but I was rejected with no feedback.
Second was kinda ridiculous: implement a stripped-down version of their website, with tests (and production-ready, of course). They claimed it could be done in 4 hours, which is probably true for all the devs that work there, since they already know exactly what's involved. I spent ~8 hours on it and submitted something that worked but had problems, which I documented thoroughly. Again, rejected with no feedback.
Second was kinda ridiculous: implement a stripped-down version of their website, with tests (and production-ready, of course). They claimed it could be done in 4 hours, which is probably true for all the devs that work there, since they already know exactly what's involved. I spent ~8 hours on it and submitted something that worked but had problems, which I documented thoroughly. Again, rejected with no feedback.
Were you interviewing in Brisbane, Australia? Sounds very similar to a recent experience of mine.
More and more it is becoming "standard policy" to give zero feedback. It's quite pathetic to be honest
I have fought the “zero feedback” policies at companies going back ~12 years now. I think it’s terrible but I always lose.
But here is why. For every personal story you have of working with a dev that can’t program, your recruiting staff has 3 of bad/lawyer invoking instances of dealing with crazy candidates. They are being honest with you when they tell you any other policy is too expensive.
But here is why. For every personal story you have of working with a dev that can’t program, your recruiting staff has 3 of bad/lawyer invoking instances of dealing with crazy candidates. They are being honest with you when they tell you any other policy is too expensive.
This is why we use a timer on our take home exercise.
You are only supposed to spend 6 hours on it, we give a 24 hours timer so you are not too pressured for time (you can take breaks, have diner, a walk, etc) .
So far it works pretty well.
I have not yet seen any applicant writing tests. Just delivering the project with the requested features and acceptable code (no memory leaks, crashes, unreadable mess, etc) is more than enough to go to the next step.
If you demonstrate enough proficiency during this step, we will probably skip most tech questions and focus on fit in the next step.
You are only supposed to spend 6 hours on it, we give a 24 hours timer so you are not too pressured for time (you can take breaks, have diner, a walk, etc) .
So far it works pretty well.
I have not yet seen any applicant writing tests. Just delivering the project with the requested features and acceptable code (no memory leaks, crashes, unreadable mess, etc) is more than enough to go to the next step.
If you demonstrate enough proficiency during this step, we will probably skip most tech questions and focus on fit in the next step.
This seems like a good happy medium type of take-home challenge. Out of curiosity, how many applicants have you had?
Wow, I don't know, I would say 5 challenges / week.
That's our third step though, first our recruiters filter applications (I have no idea how many people are funneled out at this point)
Then we have a 30 minutes phone call to discuss experience and presell the company. We also try to answer all their questions at this point and explain the challenge. Something along the lines of : "so the next step will be a take home test. We just want to see how you code, don't put too much pressure on you, just do your best in the time given"
Historically we have only been looking for pretty senior engineers, although we are relaxing this a bit.
The company has also been sponsoring a lot of visas (even with very interesting salaries, senior mobile engineers are hard to recruit .. most of them are already happy where they are), I think that take home are a good way to handle this case.
The applicant can take the exercise whenever he wants (the only limitation is that we might not be available to answer their questions in the middle of the night) and shows us what they can do.
That's our third step though, first our recruiters filter applications (I have no idea how many people are funneled out at this point)
Then we have a 30 minutes phone call to discuss experience and presell the company. We also try to answer all their questions at this point and explain the challenge. Something along the lines of : "so the next step will be a take home test. We just want to see how you code, don't put too much pressure on you, just do your best in the time given"
Historically we have only been looking for pretty senior engineers, although we are relaxing this a bit.
The company has also been sponsoring a lot of visas (even with very interesting salaries, senior mobile engineers are hard to recruit .. most of them are already happy where they are), I think that take home are a good way to handle this case.
The applicant can take the exercise whenever he wants (the only limitation is that we might not be available to answer their questions in the middle of the night) and shows us what they can do.
> but they want tests and for me to treat it like a real production work
When hiring back-end devs, I explicitly make it clear I want at least some tests -- and the more the better. Of course I don't expect 100% test coverage but writing tests shows that 1) you understand best practices 2) you know how to write testable (readable, maintainable, clean and clear) code. If candidates chose just one module and tested that well it would be enough to get the point across.
Seeing how people write their tests is also quite important for me -- if instead of using factories (or Faker or some other strong dummy data app) they instead decide to hardcode usernames and passwords, it's a huge red flag.
When hiring back-end devs, I explicitly make it clear I want at least some tests -- and the more the better. Of course I don't expect 100% test coverage but writing tests shows that 1) you understand best practices 2) you know how to write testable (readable, maintainable, clean and clear) code. If candidates chose just one module and tested that well it would be enough to get the point across.
Seeing how people write their tests is also quite important for me -- if instead of using factories (or Faker or some other strong dummy data app) they instead decide to hardcode usernames and passwords, it's a huge red flag.
> 1) you understand best practices
Tests prove no such thing. Best practices aren't about output, they are about inputs. Tests determine if the expected output is the real output.
> 2) you know how to write testable (readable, maintainable, clean and clear) code.
Again, no they don't. I can test unreadable, unmaintainable dirty and unclear code.
> if instead of using factories (or Faker or some other strong dummy data app) they instead decide to hardcode usernames and passwords, it's a huge red flag.
No it doesn't. It shows that it's an arbitrary take home assignment that already took a week and they don't want to use yet another library to generate data for it.
Honestly, you put so much emphasis on testing which is odd, because it's the easiest, most mundane thing to write. It provides nothing but "busy" work and fails to test the most important aspect of any candidate: can they problem solve.
Tests prove no such thing. Best practices aren't about output, they are about inputs. Tests determine if the expected output is the real output.
> 2) you know how to write testable (readable, maintainable, clean and clear) code.
Again, no they don't. I can test unreadable, unmaintainable dirty and unclear code.
> if instead of using factories (or Faker or some other strong dummy data app) they instead decide to hardcode usernames and passwords, it's a huge red flag.
No it doesn't. It shows that it's an arbitrary take home assignment that already took a week and they don't want to use yet another library to generate data for it.
Honestly, you put so much emphasis on testing which is odd, because it's the easiest, most mundane thing to write. It provides nothing but "busy" work and fails to test the most important aspect of any candidate: can they problem solve.
> Tests prove no such thing. Best practices aren't about output, they are about inputs. Tests determine if the expected output is the real output
Testing in and of itself is best practices. Having familiarity with and a strong understanding of how and what to test is extremely important for any modern development cycle.
> Again, no they don't. I can test unreadable, unmaintainable dirty and unclear code.
Except it's significantly more difficult.
> No it doesn't. It shows that it's an arbitrary take home assignment that already took a week and they don't want to use yet another library to generate data for it
If you don't have a strong dummy data suite to test against, then you're not testing properly. This actually justifies exactly what I'm talking about.
> Honestly, you put so much emphasis on testing which is odd, because it's the easiest, most mundane thing to write. It provides nothing but "busy" work and fails to test the most important aspect of any candidate: can they problem solve.
If it's mundane and easy then it should just be done. What I've found is that most can't.
If the candidate pool was small, then I'd take less of an issue to it. But if 50 people come in and they can all problem solve, then I'm going to become more interested in who can solve problems elegantly.
Testing in and of itself is best practices. Having familiarity with and a strong understanding of how and what to test is extremely important for any modern development cycle.
> Again, no they don't. I can test unreadable, unmaintainable dirty and unclear code.
Except it's significantly more difficult.
> No it doesn't. It shows that it's an arbitrary take home assignment that already took a week and they don't want to use yet another library to generate data for it
If you don't have a strong dummy data suite to test against, then you're not testing properly. This actually justifies exactly what I'm talking about.
> Honestly, you put so much emphasis on testing which is odd, because it's the easiest, most mundane thing to write. It provides nothing but "busy" work and fails to test the most important aspect of any candidate: can they problem solve.
If it's mundane and easy then it should just be done. What I've found is that most can't.
If the candidate pool was small, then I'd take less of an issue to it. But if 50 people come in and they can all problem solve, then I'm going to become more interested in who can solve problems elegantly.
Directed at you and the parent. A better way to structure this is ask for at least one integration test, then tell the candidate that testing strategy and methodology will be discussed in the review.
I had a coding test, not take home, where I was able to code on my own on a simple problem for about 45 minutes for each session then have a code review. We did this three times over three hours on increasingly harder problems.
This exact same format could be done remotely.
I had a coding test, not take home, where I was able to code on my own on a simple problem for about 45 minutes for each session then have a code review. We did this three times over three hours on increasingly harder problems.
This exact same format could be done remotely.
> they instead decide to hardcode usernames and passwords, it's a huge red flag.
Why? That seems like an oddly specific thing to be a _huge_ red flag. Are you expecting random data to actually surface bugs - ie: making your tests non-deterministic? Or is there some other benefit you are looking for to make it such an important criteria?
Why? That seems like an oddly specific thing to be a _huge_ red flag. Are you expecting random data to actually surface bugs - ie: making your tests non-deterministic? Or is there some other benefit you are looking for to make it such an important criteria?
Especially for something as throw away as a take home coding challenge.
It may be a throwaway for the candidate, for the employer it's $XXX,000 a year and potentially ending the search for better, more suited candidates.
If taking an extra an hour of time to sell your work is "throwaway" then that person is either shotgun applying to jobs (and not being hired for a reason) or they're not taking their application seriously.
If taking an extra an hour of time to sell your work is "throwaway" then that person is either shotgun applying to jobs (and not being hired for a reason) or they're not taking their application seriously.
That's fine, just understand that is a technique for finding "growing" developers, and still not a great one for that. If that's what you are looking for, that's fine. If you want tops, this isn't the way to go. Unless you've already hired someone that knows a top, you probably won't get them anyway.
Think about it this way. It takes about 2 hours to learn how to write unit tests for someone who has never done it. Why make something as trivial so important in your interview process? I understand unit tests are important, but they aren't difficult.
Think about it this way. It takes about 2 hours to learn how to write unit tests for someone who has never done it. Why make something as trivial so important in your interview process? I understand unit tests are important, but they aren't difficult.
> It takes about 2 hours to learn how to write unit tests for someone who has never done it.
If it's really that trivial, the developer should be doing it so routinely that it should take less than 15 minutes to write unit tests for the take home exercise. Your statement literally demolishes any excuse for not doing it.
If it's really that trivial, the developer should be doing it so routinely that it should take less than 15 minutes to write unit tests for the take home exercise. Your statement literally demolishes any excuse for not doing it.
Good unit tests themselves, if you have good coverage, usually take longer than writing the code. But you knew that right?
> Good unit tests themselves, if you have good coverage, usually take longer than writing the code. But you knew that right?
All too well, my friend. That is why the employer is justified in asking for a demonstration of the ability to write good unit tests, as you have demonstrated by your own admission.
All too well, my friend. That is why the employer is justified in asking for a demonstration of the ability to write good unit tests, as you have demonstrated by your own admission.
He said if the user didn't use factories for test data, it would be a huge red flag.
According to him, if I didn't setup a factory, when doing a take home assignment, then I'm not competent to write unit tests. I disagree.
So many people interview poorly. It's gotten worse every time I read about it. They're pushing away talent that way. I know people are stuck in their ways, but it's a bad idea unless you are are just looking for juniors and people desperate enough to go through that. I'm neither.
I mean don't people realize it's a seller's market?
According to him, if I didn't setup a factory, when doing a take home assignment, then I'm not competent to write unit tests. I disagree.
So many people interview poorly. It's gotten worse every time I read about it. They're pushing away talent that way. I know people are stuck in their ways, but it's a bad idea unless you are are just looking for juniors and people desperate enough to go through that. I'm neither.
I mean don't people realize it's a seller's market?
Then maybe communicate that salary amount to the candidate before so they can also decide if it is worth their time or not.
It shows a lack of familiarity with the tooling. It's actually quicker to write something that prototypes the model you're testing -- as little as a few lines of code -- that eventually saves a lot of repetition.
That prototyping later becomes extremely useful for building local databases -- a strong factory suite means I can rebuild my local database and have it completely populated with randomized dummy data. Changing locales on some packages means I can have a local environment populated with data tailored for a completely different country and language, for example.
Re: surfacing bugs -- in a lot of cases yes. And a lot of bugs have been found this way.
That prototyping later becomes extremely useful for building local databases -- a strong factory suite means I can rebuild my local database and have it completely populated with randomized dummy data. Changing locales on some packages means I can have a local environment populated with data tailored for a completely different country and language, for example.
Re: surfacing bugs -- in a lot of cases yes. And a lot of bugs have been found this way.
It looks like you're mistaking "good programming" for "programming the same way I do".
If a developer can test, that's a good fundamental skill. If they don't use a factory, that says nothing about their fundamentals – it's something you can easily tell someone who knows how to test to go learn & do once they're on the job.
If a developer can test, that's a good fundamental skill. If they don't use a factory, that says nothing about their fundamentals – it's something you can easily tell someone who knows how to test to go learn & do once they're on the job.
Examples like this are why I think take-home challenges are a better way to assess a candidate's coding skills vs. their problem solving skills/how they think.
Our strategy is to save the take-home for after the in person interview, and to pay you for the time (up to the limit we expect), then a follow up to talk about the work.
I'm about to start applying to new positions and it sounds like I won't get the same courtesy, but only you can decide if your enthusiasm for the position matches the opportunity cost of the interview process.
I'm about to start applying to new positions and it sounds like I won't get the same courtesy, but only you can decide if your enthusiasm for the position matches the opportunity cost of the interview process.
Can you really justify my contractor rates though? My time isn't cheap, and I don't work until the down payment has cleared.
All a take home test does is eliminate your strongest candidates, since they generally won't bother. It has to be a truly extraordinary position before anyone with significant experience will waste their time trying to compete in this market. Keep in mind my opportunity cost while job seeking is networking. Every minute I waste on someone's takehome is time I could be tayloring my resume to another position or emailing with prospective employers, which makes them all the more expensive to waste time on.
All a take home test does is eliminate your strongest candidates, since they generally won't bother. It has to be a truly extraordinary position before anyone with significant experience will waste their time trying to compete in this market. Keep in mind my opportunity cost while job seeking is networking. Every minute I waste on someone's takehome is time I could be tayloring my resume to another position or emailing with prospective employers, which makes them all the more expensive to waste time on.
shrug offering pay isn't meant as an insult, but to show we value your time. If we can't afford your hourly rates, we probably can't afford to offer you a salary you'd be happy with.
Really?
Considering I'll have to pay all taxes, medicare, ssi, etc. on all self-employment income and that the time I spend working on your challenge is time I'm not spending chasing other business, the contractor rate that I'm going to charge you is likely going to be around 300-400% of what I expect my salary to be for that time.
If you can't afford that, I'll probably think you're either exploitative or delusional or both.
But then again, your 6 hour programming assignment is worth about $1500+ of my time.
We take a salary because we want to offload risks and time spent doing sales to source our income, but we shouldn't have to do throwaway work for free to get that. Sounds too much like 'unpaid internship' to me.
Considering I'll have to pay all taxes, medicare, ssi, etc. on all self-employment income and that the time I spend working on your challenge is time I'm not spending chasing other business, the contractor rate that I'm going to charge you is likely going to be around 300-400% of what I expect my salary to be for that time.
If you can't afford that, I'll probably think you're either exploitative or delusional or both.
But then again, your 6 hour programming assignment is worth about $1500+ of my time.
We take a salary because we want to offload risks and time spent doing sales to source our income, but we shouldn't have to do throwaway work for free to get that. Sounds too much like 'unpaid internship' to me.
Note that in some jurisdictions, the candidates cannot take that pay without reporting it to their existing employers.
I completely agree with your criticism but I disagree with your conclusion.
The fact that some applicants will spend ridiculous amounts of time on a challenge is a critical flaw. It doesn't mean that take-home challenges are worthless, though. It just means they need a functional mechanism to enforce time limits equally on all candidates.
The fact that some applicants will spend ridiculous amounts of time on a challenge is a critical flaw. It doesn't mean that take-home challenges are worthless, though. It just means they need a functional mechanism to enforce time limits equally on all candidates.
> Some people spend an entire week on the coding challenge, which really set the bar unrealistically high.
Simple, cater the challenge so that's not possible.
Some companies are just bad at conducting interviews, that's the real tell and not take-home challenges.
I've had terrible in-person technical interviews, but I'm not going to reject the entire concept of in-person technical interviews because I've had a couple bad experiences.
If someone doesn't want to participate in a take-home coding challenge then that's their prerogative, but not everyone thinks they are a bad idea. If I were requesting a take-home challenge and a candidate I liked said they will not proceed if they have to do one, then I would probably make an exception. But that's me working at a startup where I have full interview control.
Simple, cater the challenge so that's not possible.
Some companies are just bad at conducting interviews, that's the real tell and not take-home challenges.
I've had terrible in-person technical interviews, but I'm not going to reject the entire concept of in-person technical interviews because I've had a couple bad experiences.
If someone doesn't want to participate in a take-home coding challenge then that's their prerogative, but not everyone thinks they are a bad idea. If I were requesting a take-home challenge and a candidate I liked said they will not proceed if they have to do one, then I would probably make an exception. But that's me working at a startup where I have full interview control.
Thanks! I definitely understand that many people are opposed to take-home challenges because they can be very time consuming. For me, I use them as a learning opportunity, so even though they are not necessarily going to be used or maintained in the future, they can be a great way to practice or get better at a specific language or tech stack.
While this might be true, most developers (in my locale at least) fill their GitHub with little projects meant to fit this exact purpose. I would 100% consider someone with some semblance of a GitHub over someone who submitted a well done takehome with no GitHub. That's why I don't bother with them. I would rather see that you are at least trying to work on yourself as a developer outside of interviews and work. It shows initiative, dedication and a host of other great attributes that I just don't see from a takehome. I'm not saying there aren't great coders without GitHub accounts, but I would for sure be asking why not. I think a better use of time is to do "production" code on a whiteboard. FizzBuzz in person, if done properly, can tell you way more than any take home. Start with basic FizzBuzz or any other basic interview question and make them write it functionally and tesably and mock out some sample tests. In person, this will tell you a ton about the candidate and isn't super stressful for anyone because it's a well known baseline interview question. It also makes the interviewee feel like the company is investing as much time in the interview process as they are.
Many of us signed contracts where 100% of the development work we do belongs to our company even in our off time. Myself included (though it's only if the work is in the same industry -- at the discretion of my employer of course). Some of the best companies hiring the best engineers do this. You are seriously limiting your pool if you favor candidates who spend their off time contributing to their GitHub. It's a bad bias to have.
Worse still, I've interviewed people who have put their employers' work on their personal GitHub, publicly and without permission, solely because of this reason. I have serious ethics/intelligence questions about those people and they become a strong no-hire every time I see it.
Worse still, I've interviewed people who have put their employers' work on their personal GitHub, publicly and without permission, solely because of this reason. I have serious ethics/intelligence questions about those people and they become a strong no-hire every time I see it.
I guess we will just have to disagree. What works for me may not work for others. I would never sign a contract that sells all my personal work to the company I only get paid 9 to 5 for, and I don't know anyone who has. Also, it's pretty evident when someone posts their companies code to their GitHub. Simple questions and looking at the code can dispel that. If they do manage to squeak by and pass all the interviews, I don't see how a takehome would have made it any harder anyway.
The motivation is that you're showing off your skills with this code. You want to deliver the code that, in your opinion, is the best possible for solving the given task.
> which will never be maintained because its a take-home assignment
You will most likely be asked to add a feature or two to it in the followup in person interview.
> which will never be maintained because its a take-home assignment
You will most likely be asked to add a feature or two to it in the followup in person interview.
I've did a few of these take home challenges during the interviewing process.
All of mines have had a time limit to them of an hour. The process has always been
1) arrange a time with the recruiter when I will be free for an hour, e.g.lets say 1PM.
2) recruiter emails me the challenge at around 12:58. I get the challenge and have to code the solution and have it back to the recruiter by 2PM.
Then at the face to face interview we've went through a code review of my code and talked about why I chose to do somethigns the way I did, if I had to more time what would I do differently, was there anything in my solution that I wasn't happy with etc.
Seems to work reasonably well.
There is no way on earth I would be dedicating a weekend or more to an interview task.
All of mines have had a time limit to them of an hour. The process has always been
1) arrange a time with the recruiter when I will be free for an hour, e.g.lets say 1PM.
2) recruiter emails me the challenge at around 12:58. I get the challenge and have to code the solution and have it back to the recruiter by 2PM.
Then at the face to face interview we've went through a code review of my code and talked about why I chose to do somethigns the way I did, if I had to more time what would I do differently, was there anything in my solution that I wasn't happy with etc.
Seems to work reasonably well.
There is no way on earth I would be dedicating a weekend or more to an interview task.
That's pretty good reason if that's true but setting that expectation would be useful. I've also never had that happen to me.
I think basing the followup interview on the homework task is pretty standard.
You can always have someone else do the "homework" for you, but when you keep working on the same code base you have to show a real understanding of both the problem and your code, and it's quite close to a real work situation.
In hindsight I agree that telling people that will happen would level the playing field a bit.
You can always have someone else do the "homework" for you, but when you keep working on the same code base you have to show a real understanding of both the problem and your code, and it's quite close to a real work situation.
In hindsight I agree that telling people that will happen would level the playing field a bit.
There are several possible reasons why you didn't include any tests in a take home exercise, and they are all bad.
1) You ran out of time to write tests.
This might be the least bad reason. You can probably talk your way out of this by explaining what tests you would have written if you had the time.
2) You don't know how to write tests.
This shows that you have only worked at places the don't do testing and that you haven't taken the initiative to learn about testing on your own.
3) You know how to write tests, but you don't think they are needed.
You could probably tell the interviewer why tests were not important for the particular exercise. But I have never seen a take home exercise that doesn't require a basic algorithm and some tricky conditional business logic. Both of which would benefit from unit tests.
4) The code you wrote isn't testable.
This is probably the worst reason. Everything is in one or two giant functions and you can't inject dependencies for testing without refactoring everything. This will frighten your interviewer.
1) You ran out of time to write tests.
This might be the least bad reason. You can probably talk your way out of this by explaining what tests you would have written if you had the time.
2) You don't know how to write tests.
This shows that you have only worked at places the don't do testing and that you haven't taken the initiative to learn about testing on your own.
3) You know how to write tests, but you don't think they are needed.
You could probably tell the interviewer why tests were not important for the particular exercise. But I have never seen a take home exercise that doesn't require a basic algorithm and some tricky conditional business logic. Both of which would benefit from unit tests.
4) The code you wrote isn't testable.
This is probably the worst reason. Everything is in one or two giant functions and you can't inject dependencies for testing without refactoring everything. This will frighten your interviewer.
Thanks for sharing, perhaps my workflow could be improved. Here's how I usually work:
1. Build a prototype of the feature I need to build
2. After I have a prototype, I start to understand the complexities of the project and I start thinking about how to modularize my code (the functions I would need, how the frontend-backend interacts together).
3. I rebuild the feature with TDD
For take-home assignments, I stop after step 2 because step 3 is the most time consuming part.
I'm pretty open minded about this, but with 5-6 years of development most take home assignments are so trivial and so self-contained.
Perhaps its really the take-home assignment that needs to change. For example, heres the sort of prompt that would motivate me to write good tests:
"Here's a private repository that every candidate we have hired has been building. We would like you to add a feature. Please build feature X on top of our existing code -> Link to repository that has an application with simple testing infrastructure"
Now the take-home assignment has more value and actually gauges how well the candidate would do in an actual working environment. When the candidate gets hired, his feature actually gets merged into the assignment and a new feature is assigned to new candidates.
1. Build a prototype of the feature I need to build
2. After I have a prototype, I start to understand the complexities of the project and I start thinking about how to modularize my code (the functions I would need, how the frontend-backend interacts together).
3. I rebuild the feature with TDD
For take-home assignments, I stop after step 2 because step 3 is the most time consuming part.
I'm pretty open minded about this, but with 5-6 years of development most take home assignments are so trivial and so self-contained.
Perhaps its really the take-home assignment that needs to change. For example, heres the sort of prompt that would motivate me to write good tests:
"Here's a private repository that every candidate we have hired has been building. We would like you to add a feature. Please build feature X on top of our existing code -> Link to repository that has an application with simple testing infrastructure"
Now the take-home assignment has more value and actually gauges how well the candidate would do in an actual working environment. When the candidate gets hired, his feature actually gets merged into the assignment and a new feature is assigned to new candidates.
[deleted]
This is a really cool idea - because working with code written by other people is usually a large part of any job, this type of assignment could also test for that. For newer devs, I've seen anything from simple Tic Tac Toe to being given a starter app with a general set of requirements to fulfill.
Or most likely:
5) Your interview process is ridiculous, and I refuse to do more than the bare minimum when I am not even guaranteed an interview in exchange for my time investment.
I am an experienced programmer. I refuse to do take-home tests as a pre-screen for an actual interview, unless I know for certain that a position is one that I would want to take (e.g. I know someone who works there, and I know that it's a dream position. This basically never happens.) Moreover, if I have done a full interview loop and you insist on screening me based on inanity such as "he didn't write tests for this time-limited toy assignment", I will gladly move on to the next company that isn't ridiculous.
Take-home tests are screening devices. If you draw inferences other than "this candidate can/cannot write code" from a take-home test, you are using them incorrectly.
5) Your interview process is ridiculous, and I refuse to do more than the bare minimum when I am not even guaranteed an interview in exchange for my time investment.
I am an experienced programmer. I refuse to do take-home tests as a pre-screen for an actual interview, unless I know for certain that a position is one that I would want to take (e.g. I know someone who works there, and I know that it's a dream position. This basically never happens.) Moreover, if I have done a full interview loop and you insist on screening me based on inanity such as "he didn't write tests for this time-limited toy assignment", I will gladly move on to the next company that isn't ridiculous.
Take-home tests are screening devices. If you draw inferences other than "this candidate can/cannot write code" from a take-home test, you are using them incorrectly.
As an experienced programmer, what does your ideal interview process look like? Do you think your opinion holds true for junior developers as well?
The ideal hiring process imo is actually a training process that is less focused on 'finding the best' but rather focused on helping their local community become great engineers. Companies fund their own bootcamps and pay students $15/hr (with no engineering experience). Students learn and help each other how to code (following a structured curriculum) with mentorship by engineers from the company. When students have finished the basics, they work on open source projects or internal projects following proper engineering process / tools that the company use. When a student has proved that he/she learned how to work well with other engineers and has built a few features, they get hired by the company as a junior engineer and this junior dev would be immediately productive.
This might seem like a crazy idea, but its actually not. The previous companies that I have worked for all offer paid volunteer hours which could go directly to this initiative. The cost of hiring is around 30k per engineer (conservatively), which could help train someone living paycheck to paycheck for an entire year.
I spent the past year and a half starting a non-profit experimenting with this idea and invested all 200k of my annual salary to paying students. The coding tools and environment is set up like a baby version of Google's engineering infrastructure. Students progress are tracked by their merge requests and the complexity of the features that they have built. Students build tools that they use on a daily basis so they are heavily eating their own dogfood. Hopefully, one day I'll be able to build my engineering team by directly hiring from this non-profit. https://garagescript.org
This might seem like a crazy idea, but its actually not. The previous companies that I have worked for all offer paid volunteer hours which could go directly to this initiative. The cost of hiring is around 30k per engineer (conservatively), which could help train someone living paycheck to paycheck for an entire year.
I spent the past year and a half starting a non-profit experimenting with this idea and invested all 200k of my annual salary to paying students. The coding tools and environment is set up like a baby version of Google's engineering infrastructure. Students progress are tracked by their merge requests and the complexity of the features that they have built. Students build tools that they use on a daily basis so they are heavily eating their own dogfood. Hopefully, one day I'll be able to build my engineering team by directly hiring from this non-profit. https://garagescript.org
Love the idea of investing in the local community. Training and mentorship is so important, and so many companies sacrifice it for other things.
I probably wouldn't do a take home test for an interview, but if you're only willing to do the bare minimum, it would probably be a better use of your time (as well as whoever is going to grade it) if you simply refused to do it.
Before I accept a take-home project, I explicitly lay out what I will do and the maximum amount of time I will spend doing it. I've walked away from companies that demanded more than I was willing to give.
I dislike take-home tests less than I dislike whiteboard interviews, so I try to be flexible, without being a pushover.
I dislike take-home tests less than I dislike whiteboard interviews, so I try to be flexible, without being a pushover.
5) You were interested in limiting the time cost to yourself related to being interviewed, expecting that the wider sqrt(t) uncertainty band in your abilities would not mess up the company's hiring procedure too badly.
In general I find them to be completely unrealistic if you're trying to find anyone good.
Someone really good will already have a full time job, a limited number of "sick days" to use for interviewing and is probably somewhere in the interview process with at least 3 other companies -- probably closer to 5 or 7. Then they might have family responsibilities on top of that.
I budget 4, maybe 6 hours if I really like you, on any company in the application process -- tops. Almost 100% of this time is taken up by interviews. You want me to give your company a whole weekend of my time on some chance that you might hire me? In addition to hours of interviews? That already tells me that I don't want to work for you -- especially if I haven't at least interviewed with 3 technical people already. It's one of the clearest negative signals I know of.
If I work 13 days in a row without a day off, I'm going to be a crabby asshole at my job and probably blow all of my other interviews until I recuperate. Your company isn't worth that.
Someone really good will already have a full time job, a limited number of "sick days" to use for interviewing and is probably somewhere in the interview process with at least 3 other companies -- probably closer to 5 or 7. Then they might have family responsibilities on top of that.
I budget 4, maybe 6 hours if I really like you, on any company in the application process -- tops. Almost 100% of this time is taken up by interviews. You want me to give your company a whole weekend of my time on some chance that you might hire me? In addition to hours of interviews? That already tells me that I don't want to work for you -- especially if I haven't at least interviewed with 3 technical people already. It's one of the clearest negative signals I know of.
If I work 13 days in a row without a day off, I'm going to be a crabby asshole at my job and probably blow all of my other interviews until I recuperate. Your company isn't worth that.
This is a little silly. If you’re “really good” you may be talking about signing on for a 4-5 expected commitment at $1M+ Total compensation over the course of your tenure. Million dollar contracts require due diligence. 4-5 man days between both companies is totally reasonable.
If you’re interviewing with 7 (!) companies at a time, maybe be a little more discerning about where you’re trying to work. Everybody wins if you’re specific on that point.
If you’re interviewing with 7 (!) companies at a time, maybe be a little more discerning about where you’re trying to work. Everybody wins if you’re specific on that point.
> If you’re interviewing with 7 (!) companies at a time, maybe be a little more discerning about where you’re trying to work. Everybody wins if you’re specific on that point.
Oh please, honestly. I'm not a terribly good programmer, but after 2+ years at one company, my inbox was flooded with emails from recruiters representing companies with very interesting opportunities (and like 10x as much crap). Then there's the friends and former coworkers who have been regularly asking me to come work with them.
Once I made the decision to leave my last company, I talked to about 9 companies over a 1.5 month period, and it only lasted that long because of Christmas & New Years. At least 5 of those went through multiple interview rounds and I was juggling 4 different, attractive offers before picking one. This is just from the 9 that I decided to talk to...
I live in a major market and there's a lot of demand here and in my focus areas. A lot of companies that you don't think have interesting problems actually have fascinating problems to work on once you talk to them. Probably much more so than GOOG, FB, AMZN, etc.
You talk about being discerning, but how do you know if you can deliver value until you have a conversation with them? You end up talking to that many companies at once because once you start the process with them you have to be respectful and see their process through to the end, but you don't stop interviewing while you wait for a decision.
Oh please, honestly. I'm not a terribly good programmer, but after 2+ years at one company, my inbox was flooded with emails from recruiters representing companies with very interesting opportunities (and like 10x as much crap). Then there's the friends and former coworkers who have been regularly asking me to come work with them.
Once I made the decision to leave my last company, I talked to about 9 companies over a 1.5 month period, and it only lasted that long because of Christmas & New Years. At least 5 of those went through multiple interview rounds and I was juggling 4 different, attractive offers before picking one. This is just from the 9 that I decided to talk to...
I live in a major market and there's a lot of demand here and in my focus areas. A lot of companies that you don't think have interesting problems actually have fascinating problems to work on once you talk to them. Probably much more so than GOOG, FB, AMZN, etc.
You talk about being discerning, but how do you know if you can deliver value until you have a conversation with them? You end up talking to that many companies at once because once you start the process with them you have to be respectful and see their process through to the end, but you don't stop interviewing while you wait for a decision.
This might make sense for the >$200k jobs, but you know those are specific to a few companies in a few cities. Yet the homework trend will spread to all sorts of companies with much less competitive money.
It's very interesting how even the Internet mega-giants haven't solved distributed working enough that they can avoid paying the massive premium for people to move to SF.
It's very interesting how even the Internet mega-giants haven't solved distributed working enough that they can avoid paying the massive premium for people to move to SF.
Yikes at the typos!
- 4-5 year expected commitment.
- 4-5 man-days between both parties.
Anyway, talking to a company is rather different than interviewing. Sure, explore opportunities. But no need to write a line of code until they’ve made an attractive proposition. I’d be surprised to see that happen 7 times within a week or two.
- 4-5 year expected commitment.
- 4-5 man-days between both parties.
Anyway, talking to a company is rather different than interviewing. Sure, explore opportunities. But no need to write a line of code until they’ve made an attractive proposition. I’d be surprised to see that happen 7 times within a week or two.
Just for some color: The only company that's given me a weekend challenge that ever gave me an offer was 8th Light. I've done live coding and whiteboarding with other companies and received offers, but the take home work has only been productive that one time. I've had code I submitted to these looked over by better programmers than myself and heard very few negative comments, all minor.
After some years of experience, I can't see myself doing one of these again pretty much ever.
After some years of experience, I can't see myself doing one of these again pretty much ever.
I think the only humane way to administer a take-home coding challenge is with an enforced time limit. That way everyone is on a level playing field and you're not advantaging those candidates that happen to have oodles of free time.
The author raises that point:
https://www.fullstackinterviewing.com/2018/02/02/the-ultimat...
But takes the wrong view on it in my opinion. The answer simply can't be "Just spend as long as it takes".
That point is the thing that led me to build a tool that enforces time limits on challenges via a custom git server. I used it to interview around 100 candidates at a company I was running at the time and it was an excellent approach.
I've since productised it as https://takehome.io/
If you want the benefits that take-home coding challenges can bring to your hiring process, but are concerned about treating your candidates humanely, I urge you to check it out.
The author raises that point:
https://www.fullstackinterviewing.com/2018/02/02/the-ultimat...
But takes the wrong view on it in my opinion. The answer simply can't be "Just spend as long as it takes".
That point is the thing that led me to build a tool that enforces time limits on challenges via a custom git server. I used it to interview around 100 candidates at a company I was running at the time and it was an excellent approach.
I've since productised it as https://takehome.io/
If you want the benefits that take-home coding challenges can bring to your hiring process, but are concerned about treating your candidates humanely, I urge you to check it out.
>> enforces time limits on challenges
But there is a problem. In the past when I have worked on take home challenges which usually take 3 hours or thereabouts, I have done them in 3 sessions of 1 hour each in the evening after work:
1 hour: think about the problem in detail making some notes for scenarios, edge cases, high level API design etc 2nd hour: first rough cut working draft with some test coverage 3rd hour: polish it up with more tests, docs, a README and sending it out.
I think `takehome.io` should account for time boxing across multiple sessions, else it might be pretty much useless for working engineers who do it across time boxed sessions :-)
PS: I apologize in advance if you already have this feature, but is not so readily evident on your product home page.
But there is a problem. In the past when I have worked on take home challenges which usually take 3 hours or thereabouts, I have done them in 3 sessions of 1 hour each in the evening after work:
1 hour: think about the problem in detail making some notes for scenarios, edge cases, high level API design etc 2nd hour: first rough cut working draft with some test coverage 3rd hour: polish it up with more tests, docs, a README and sending it out.
I think `takehome.io` should account for time boxing across multiple sessions, else it might be pretty much useless for working engineers who do it across time boxed sessions :-)
PS: I apologize in advance if you already have this feature, but is not so readily evident on your product home page.
Thanks for the feedback! I appreciate your perspective on time limits.
I don't have that feature, because I would argue that a problem you attack in three 1 hour stints across a span of 72 hours is a 72 hour problem.
Writing software is so much more than just typing. I tend to do my best coding in the shower or driving, nowhere near a computer. As soon as the challenge has been loaded into your brain, the work starts and the timer starts. How much of that time you spend in front of the keyboard is up to you.
Incidentally, I also think that a time limit of longer than an hour is probably too long. Uninterrupted multi-hour blocks are pretty hard to come by. Even if the time limit given isn't enough time for you to complete a perfect, working solution to the problem, it should be enough time for you to do enough work that we've got something meaty to talk about in the next interview.
I don't have that feature, because I would argue that a problem you attack in three 1 hour stints across a span of 72 hours is a 72 hour problem.
Writing software is so much more than just typing. I tend to do my best coding in the shower or driving, nowhere near a computer. As soon as the challenge has been loaded into your brain, the work starts and the timer starts. How much of that time you spend in front of the keyboard is up to you.
Incidentally, I also think that a time limit of longer than an hour is probably too long. Uninterrupted multi-hour blocks are pretty hard to come by. Even if the time limit given isn't enough time for you to complete a perfect, working solution to the problem, it should be enough time for you to do enough work that we've got something meaty to talk about in the next interview.
This is how I do it, as a Director dn hiring manager. 4 hours max, and the exercise is entirely freeform against a set of 10 typical requirements; as a Front End Engineer candidate for my team, I want to see your opinion and execution matched to the problem at hand.
It can be done in 2 hours in the most simplistic, undeliberated way using legacy practices, but 8 if you're a perfectionist. Come in somewhere in the middle, but stop at 4 hours no matter where you're at.
I don't even require that the exercise is completed, but I want you to speak to your thought process and where you went/were going when you hit the 4 hour mark. The ability to understand our space, make good decisions, and show progress toward executing on them is ultimately what matters in the end.
It can be done in 2 hours in the most simplistic, undeliberated way using legacy practices, but 8 if you're a perfectionist. Come in somewhere in the middle, but stop at 4 hours no matter where you're at.
I don't even require that the exercise is completed, but I want you to speak to your thought process and where you went/were going when you hit the 4 hour mark. The ability to understand our space, make good decisions, and show progress toward executing on them is ultimately what matters in the end.
That fits exactly with my method and thus the ethos of https://takehome.io
I don't mean to shill, but I think it'll be a great fit for you.
I don't mean to shill, but I think it'll be a great fit for you.
Thanks for the tip; we've been doing it in a similar fashion, and this solves the biggest variable: "How do we enforce that 4 hour maximum?"
This is really cool, thanks for sharing! I don't necessarily disagree with your point, though I do think an unrealistic time limit is even worse than no time limit. I also emphasize the importance of not taking on a challenge that is outside the scope of your ability here: https://www.fullstackinterviewing.com/2018/02/02/the-ultimat...
What are your thoughts on timed coding problems like Codility or Hackerrank?
What are your thoughts on timed coding problems like Codility or Hackerrank?
Thank-you!
My contention is that it doesn't matter if the time limit is unrealistic (within reason) so long as it's consistent.
ie: the challenge is not "Show me the best solution to this problem" but rather "Show me the best solution to this problem that you can achieve within an hour".
As long as everyone is under the same conditions, the playing field is level.
It's part of a holistic approach, too. The coding challenge is not the only part of the interview process. I suggest that there be an initial interview to assess experience, fit, explain the job, etc. Then if everyone is still interested, the coding challenge. The candidate's response to the challenge then forms the basis for a discussion in another, technically focused interview.
It's not about saying "You gave the wrong answer" but "Talk me through why you solved it this way" and "If this criteria in the challenge changed, tell me about how that might change your response"
The Codility and Hackerrank style problems are okay, but I don't like how constrained they are. I also think it's pointless having someone outside your organisation grade the results. What matters is not having a candidate who is "the best hacker" but having someone who fits what you need. If your team values robustness and debuggability above all else, you _don't_ want someone who can punch out 3 kinda-right prototypes before lunch. If you're an ideation agency, the inverse is true.
That's why takehome encourages companies to create and administer their own challenges rather than the samples. They should fit the kinds of problems your organisation faces, not contrived ones.
My contention is that it doesn't matter if the time limit is unrealistic (within reason) so long as it's consistent.
ie: the challenge is not "Show me the best solution to this problem" but rather "Show me the best solution to this problem that you can achieve within an hour".
As long as everyone is under the same conditions, the playing field is level.
It's part of a holistic approach, too. The coding challenge is not the only part of the interview process. I suggest that there be an initial interview to assess experience, fit, explain the job, etc. Then if everyone is still interested, the coding challenge. The candidate's response to the challenge then forms the basis for a discussion in another, technically focused interview.
It's not about saying "You gave the wrong answer" but "Talk me through why you solved it this way" and "If this criteria in the challenge changed, tell me about how that might change your response"
The Codility and Hackerrank style problems are okay, but I don't like how constrained they are. I also think it's pointless having someone outside your organisation grade the results. What matters is not having a candidate who is "the best hacker" but having someone who fits what you need. If your team values robustness and debuggability above all else, you _don't_ want someone who can punch out 3 kinda-right prototypes before lunch. If you're an ideation agency, the inverse is true.
That's why takehome encourages companies to create and administer their own challenges rather than the samples. They should fit the kinds of problems your organisation faces, not contrived ones.
Good point about consistency, which is hard to achieve with the open-ended take-home challenges. I still feel like there is a catch 22 for new developers though - if you're competing with more experienced candidates, your best solution that you can achieve within one hour will rarely be one of the better ones, and the only way to really get better is to have real experience working on a production app. And you still have to get hired to do that, hence the catch 22. So, that's why I advocate for spending more time on these challenges up front, especially for people first making the switch, because they can be used as a learning opportunity. So many companies don't have the time or just don't want to invest the time to train newer developers, which automatically eliminates hiring someone straight out of a bootcamp. Maybe someone with a CS degree and no work experience would have the same problem though.
I like what you said about the importance of companies administering their own challenges as well. I've encountered some like that and they do tend to be a better reflection of the types of challenges and problems the company is facing.
I like what you said about the importance of companies administering their own challenges as well. I've encountered some like that and they do tend to be a better reflection of the types of challenges and problems the company is facing.
As someone who's just entered the employment market after being a freelancer for a while, I have no idea how people who already have jobs can physically participate in the lengthy interview processes most jobs require these days. Ok they can take sick days, but if you're a bad liar or have an overactive conscience that's not ideal.
I totally understand the value of coding challenges, but I've been endlessly surprised at companies that expect me to expend more than 4 hours of my life interviewing for a chance at a job that will be paying maximum 50k. These are small companies, hardly the big 4. It's really an imposition.
I totally understand the value of coding challenges, but I've been endlessly surprised at companies that expect me to expend more than 4 hours of my life interviewing for a chance at a job that will be paying maximum 50k. These are small companies, hardly the big 4. It's really an imposition.
[deleted]
There's a lot of contempt for take-home coding tests in this thread. They aren't a perfect solution by any means (1), but much closer to an engineer's real responsibilities than a whiteboard session.
For this reason and this reason alone, I will continue to advocate for take-home coding challenges.
(1) IMHO, the best practical solution is a pair-programming interview using a real bug or feature the company has already solved.
For this reason and this reason alone, I will continue to advocate for take-home coding challenges.
(1) IMHO, the best practical solution is a pair-programming interview using a real bug or feature the company has already solved.
Challenges administered via pair programming can be useful if your team/company operates predominantly in that mode. If most of your actual day-to-day work is done individually, though, I think your challenge should mirror that reality.
Interviewing all candidates in a pair programming setting risks weeding out all the people who are great at the job, but get anxious in a paired interview scenario.
A job interview is already pretty stressful and anxiety inducing. Adding somebody looking over your shoulder while you type into that mix can be really rough on some people.
Interviewing all candidates in a pair programming setting risks weeding out all the people who are great at the job, but get anxious in a paired interview scenario.
A job interview is already pretty stressful and anxiety inducing. Adding somebody looking over your shoulder while you type into that mix can be really rough on some people.
I also prefer take homes. I find whiteboard interviews are much worse, because I have to relearn a lot of things in detail that aren't useful in my previous job or my next, just because these are classical whiteboard problems. Take homes are actually about job skills, not some how good are you at memorizing 100s of algorithms you should never program metric.
Whiteboarding is only good for those who are fresh out of college.
Whiteboarding is only good for those who are fresh out of college.
No. Talk to them, get a feel, if they are a fit, hire them for a 3 month contract. If they aren't any good, don't hire them. If they are really bad, cut the contract short.
You will never know what someone is capable of in a few hours. People who think that works are just fooling themselves and turning away good talent.
You will never know what someone is capable of in a few hours. People who think that works are just fooling themselves and turning away good talent.
And what of people who don't want to leave existing full time employment for a three month contract that might turn into a job if things work out well? Similarly, I doubt most people would be willing to move to a city for a temporary contract.
Usually if people are looking around, they are pretty fed up with the place they are at for one reason or another. If it's a good developer, they would know they would make the cut, so they would take it if it were a decent company with decent pay.
Also, there is no distinction between a full time job and a temporary contract in terms of survivability in the US. Most states are at-will states, and not being able to do the work is a fire-able offense. Just about everyone knows this.
Most people who have been in the business a while aren't willing to move to a new city, again unless you are willing to pay ungodly sums of money. If you are hiring right out of college and junior-mid's, this technique is better suited, but still, you're competing poorly in a seller's market.
I mean, just because you are hiring and paying a salary doesn't mean 20 other companies in your area aren't doing the same. What makes you better? Certainly not market rates and certainly not your exhaustive and mostly ineffective interview process.
You want a good person, ask your tops if they know anyone good, then try to lure them away with an offer of 1.2x - 1.5x market rates, depending on the return you expect from them. Trust your tops and don't put them through this silly process. Have the top interview them with you in the room. Spend about an hour with them to make sure they aren't some weirdo. Trust me, they'll respect you more if you respect their time.
Also, there is no distinction between a full time job and a temporary contract in terms of survivability in the US. Most states are at-will states, and not being able to do the work is a fire-able offense. Just about everyone knows this.
Most people who have been in the business a while aren't willing to move to a new city, again unless you are willing to pay ungodly sums of money. If you are hiring right out of college and junior-mid's, this technique is better suited, but still, you're competing poorly in a seller's market.
I mean, just because you are hiring and paying a salary doesn't mean 20 other companies in your area aren't doing the same. What makes you better? Certainly not market rates and certainly not your exhaustive and mostly ineffective interview process.
You want a good person, ask your tops if they know anyone good, then try to lure them away with an offer of 1.2x - 1.5x market rates, depending on the return you expect from them. Trust your tops and don't put them through this silly process. Have the top interview them with you in the room. Spend about an hour with them to make sure they aren't some weirdo. Trust me, they'll respect you more if you respect their time.
> You will never know what someone is capable of in a few hours.
I guess the goal is to fine tune your interviewing process based on how past candidates turned out.
I guess the goal is to fine tune your interviewing process based on how past candidates turned out.
I'd bet dollars to doughnuts the people doing this are collecting zero data to analyze to see if this new process improves or hurts the number of good candidates they actually hire.
The irony of it is putting processes like this is actually turning away good talent, but they wouldn't know, because they aren't doing any analysis; they just read an article somewhere.
The irony of it is putting processes like this is actually turning away good talent, but they wouldn't know, because they aren't doing any analysis; they just read an article somewhere.
Sorry about the late reply, but you are wrong about "collecting zero data". Without saying too much I can assure you at least one company does.
You can certainly do analysis on the hires you make, but it's pretty hard to analyze the hires you didn't make. Very few will come back and reinterview.
Sure you can, but my point was I'll bet most people don't. They just get this crap from an article they read because they don't think about all the ramifications, or don't think about it much at all.
For some businesses, they aren't hiring a ton of developers, so doing an analysis on 2-5 data points isn't going to yield anything significant.
Why would the introduce such a radical break from traditional interviewing without testing to see if it works?
I agree. I think take-home coding challenges are great. I have conducted take-home challenges and taken them. To me this does a better job measuring actual competency than a white board challenge.
I get that other developers get paid well to program so a lot of them are put-off by doing their job for "free." However, There's a simple solution of which has already been mentioned: cater the challenge to be completed within a few hours. Time-lock it and urge the applicants to only spend X number of hours on the challenge.
Like I have mentioned in another comment, I've been part of in-person interviews that lasted all day on top of an HR and technical phone interview. If I could cut down the in-person interview to just "personality fit" then that would be my preference.
I get that other developers get paid well to program so a lot of them are put-off by doing their job for "free." However, There's a simple solution of which has already been mentioned: cater the challenge to be completed within a few hours. Time-lock it and urge the applicants to only spend X number of hours on the challenge.
Like I have mentioned in another comment, I've been part of in-person interviews that lasted all day on top of an HR and technical phone interview. If I could cut down the in-person interview to just "personality fit" then that would be my preference.
Two take home "challenges" I've done in the recent past stand out for me:
1) I spent 6 hours on it. I didn't do some of the "optional" parts (unit tests specifically) because I felt like 6 hours was enough to "show I can program" i.e. what it was supposed to do worked. Result: I got grilled over why the optional parts weren't there and didn't get an offer.
2) I spent 15 hours on it. I did everything I could think of on it including 50ish unit tests and more than what was asked for. Result: I got and accepted the job. (note: I had laid out my salary requirements before hand).
Is this "great"? I have an active github/open source profile. To me in both cases they basically wanted to waste my time and mental effort, and see if I would do it. That.. can't be ideal.
1) I spent 6 hours on it. I didn't do some of the "optional" parts (unit tests specifically) because I felt like 6 hours was enough to "show I can program" i.e. what it was supposed to do worked. Result: I got grilled over why the optional parts weren't there and didn't get an offer.
2) I spent 15 hours on it. I did everything I could think of on it including 50ish unit tests and more than what was asked for. Result: I got and accepted the job. (note: I had laid out my salary requirements before hand).
Is this "great"? I have an active github/open source profile. To me in both cases they basically wanted to waste my time and mental effort, and see if I would do it. That.. can't be ideal.
It's a good test to see if you'll balk at doing the kinds of shit they're going to throw at you once you're hired.
Oh, you had dinner plans tonight? You want to see your kids before they go to bed? Sorry, our engineering team generally works 11am-8pm. You need to be in the office.
Oh, you had dinner plans tonight? You want to see your kids before they go to bed? Sorry, our engineering team generally works 11am-8pm. You need to be in the office.
These are good data points. I think the amount of time you should spend often depends on where you are in your career.
I know many companies don't explicitly require tests, but for them if a candidate submits a take-home challenge without them, its an automatic rejection. Do you think that if you had included tests in that first challenge that you would have gotten an offer?
I know many companies don't explicitly require tests, but for them if a candidate submits a take-home challenge without them, its an automatic rejection. Do you think that if you had included tests in that first challenge that you would have gotten an offer?
Did either interview time lock the challenges?
Part of what I like about take-home challenges is that you can do them on your own time without anyone watching over your shoulder. Because in-person technical interviews tend to last for such a long time, they can be incredibly draining. The candidate's experience really depends on the interviewer's ability as well.
Time suggestions are usualy completely off the mark. Four hours turn into 20 for anything resembling production code (which is what companies are interested in).
I like the idea of take home projects but it shouldn't come right after a phone screen or even the first real interview. More like as close to the last step as possible.
And then there's the question of what other professions ask candidates to work X hours for free. Let's assume our field is too special and not get too deep into the rabbit role ;)
I like the idea of take home projects but it shouldn't come right after a phone screen or even the first real interview. More like as close to the last step as possible.
And then there's the question of what other professions ask candidates to work X hours for free. Let's assume our field is too special and not get too deep into the rabbit role ;)
In my view successful take home assignments have 2 objective measures of if they are likely to be successful:
- did every dev at the company pass it at some point? (If so you should know fairly accurately how long it takes)
- does it replace the majority of the interview time? (If not you probably don’t actually get why it’s important)
- did every dev at the company pass it at some point? (If so you should know fairly accurately how long it takes)
- does it replace the majority of the interview time? (If not you probably don’t actually get why it’s important)
I can vouch for this. A simple 2 hour script turned into a 2 day affair since production quality code was expected. Thankfully the assignment was spread over two weeks.
I agree that time suggestions are usually way off, which can be a positive and a negative. On one hand, you can spend a ton of time to go above and beyond, while on the other hand, you may have other priorities. I think if you're really excited about a job or company, you can show that by putting a lot of effort in.
That inherently biases the interview process away from people that have additional commitments in life. You're going to end up preferentially treating those that are currently unemployed versus people that have jobs, for example.
Also, you're likely to be homogenizing your applicant base. Expect to see many more young people and recent grads versus people with kids.
Also, you're likely to be homogenizing your applicant base. Expect to see many more young people and recent grads versus people with kids.
I agree with your point. I'll add that for new developers and those without the traditional CS background (both of whom are my target audience for this guide) spending the extra time is a learning experience for them that can make them better developers.
Ultimately, it's not the applicant's choice what the parameters of the interview process are (I commend you for working to make it better) - given that, why shouldn't an applicant try to give themselves an edge in the process?
Ultimately, it's not the applicant's choice what the parameters of the interview process are (I commend you for working to make it better) - given that, why shouldn't an applicant try to give themselves an edge in the process?
You're completely right, Jane. I was viewing this pretty myopically from the hiring side as that's where I usually sit. Very true that candidates usually need to play somebody else's game, regardless of whether it's rigged.
"Work X hours for free" isn't limited to take-home assignments. I've been part of interviews that span multiple days and many hours.
Here is the breakdown of the worst one I've been part of:
* Initial phone screening with HR (~1 hr) * Technical interview over phone (~1-2 hr) * In person technical interview (~8 hr)
I kid you not, I had to take an entire day off work to be part of a grueling in-person interview where I met multiple teams.
This is also not taking into account the amount of time I had to exchange emails.
I would take a take-home assignment in the comfort of my own home, spanning 16 hours instead of an all-day in person interview.
Here is the breakdown of the worst one I've been part of:
* Initial phone screening with HR (~1 hr) * Technical interview over phone (~1-2 hr) * In person technical interview (~8 hr)
I kid you not, I had to take an entire day off work to be part of a grueling in-person interview where I met multiple teams.
This is also not taking into account the amount of time I had to exchange emails.
I would take a take-home assignment in the comfort of my own home, spanning 16 hours instead of an all-day in person interview.
I had a Meltdown ;-) after an exactly similar process. Hmm, what company was that?
>Time suggestions are usualy completely off the mark.
This is why you put a timer on the take home, it is better for everybody involved.
This is why you put a timer on the take home, it is better for everybody involved.
A take-home coding challenge will bias your pool of applicants towards relatively junior overachievers who are barely skilled enough to complete your challenge.
More senior and skilled people will decline to participate, because they know they can get a better return using the time otherwise spent on the coding challenge to network for more and easier opportunities.
So now you've done a great job of raising the floor of your interviews for little direct cost, but you've also lowered the ceiling.
More senior and skilled people will decline to participate, because they know they can get a better return using the time otherwise spent on the coding challenge to network for more and easier opportunities.
So now you've done a great job of raising the floor of your interviews for little direct cost, but you've also lowered the ceiling.
> A take-home coding challenge will bias your pool of applicants towards relatively junior overachievers who are barely skilled enough to complete your challenge.
I completely disagree with this assessment and I'll even reverse it: turning down a job opportunity simply because they requested a take-home challenge could shut you out of good opportunities.
Furthermore, no matter the interview style, there's going to be a bias.
Some people are great at white boarding, so they naturally want to take those as well as conduct them. Some interviews are all about personality fit, great, everyone will work together well.
I'm a senior engineer and would gladly accept a take-home challenge.
Ultimately as an interviewee I'd like to spend the least amount of time demonstrating the most amount of my abilities while at the same time getting a firm grasp of the company and how they conduct themselves. I firmly believe that take-home coding challenges could play a part in that balancing act between measuring competency and not wasting my time.
I completely disagree with this assessment and I'll even reverse it: turning down a job opportunity simply because they requested a take-home challenge could shut you out of good opportunities.
Furthermore, no matter the interview style, there's going to be a bias.
Some people are great at white boarding, so they naturally want to take those as well as conduct them. Some interviews are all about personality fit, great, everyone will work together well.
I'm a senior engineer and would gladly accept a take-home challenge.
Ultimately as an interviewee I'd like to spend the least amount of time demonstrating the most amount of my abilities while at the same time getting a firm grasp of the company and how they conduct themselves. I firmly believe that take-home coding challenges could play a part in that balancing act between measuring competency and not wasting my time.
I got my last 2 jobs after take home challenges (and I am a senior engineer).
The last take home was a 6 hours exercise. I did not really see it at an huge opportunity cost.
I would have trained 10 times that for a whiteboard entirely unrelated to my day to day job.
The last take home was a 6 hours exercise. I did not really see it at an huge opportunity cost.
I would have trained 10 times that for a whiteboard entirely unrelated to my day to day job.
6 hours seems too much to me. I would put maximum I am willing to spend at 4 hours.
ah, funnily enough we have recently reduced the scope of the exercise and asked only for 4 hours. At the same time we are now looking for less senior engineers.
We are also looking at ways we can make this exercise easier for everybody by providing a template with some of the basic boilerplate already written for the applicant.
It is difficult to find a happy medium between a trivial exercise and something you can work on for hundred of hours but I think we have something fair.
I find it interesting that you put your limit at 4 hours. It is not that much smaller than 6. Anecdotally 6 hours is the maximum timing .. if an aplicant said they only had 4 hours that day but delivered something solid in that time frame I would vote for them.
We are also looking at ways we can make this exercise easier for everybody by providing a template with some of the basic boilerplate already written for the applicant.
It is difficult to find a happy medium between a trivial exercise and something you can work on for hundred of hours but I think we have something fair.
I find it interesting that you put your limit at 4 hours. It is not that much smaller than 6. Anecdotally 6 hours is the maximum timing .. if an aplicant said they only had 4 hours that day but delivered something solid in that time frame I would vote for them.
> I find it interesting that you put your limit at 4 hours. It is not that much smaller than 6.
4 hours means that I can do it within half a day on weekend and then spend the rest of the day with something else - going somewhere. At minimum, I can send kids outside with someone and be 4 hours alone at home to focus. Or I can do it 2 hours one evening and 2 hours another one - two days.
6 hours means that I spent most day with it. Expecting them to be outside 6 hours in row is a lot and so is blocking one room for that long. It would also mean 3 evenings not just 2.
> At the same time we are now looking for less senior engineers.
That is cool. (Really, all too many companies like to pretend that juniors don't exists or cant learn or just should magically become seniors instantly. Meanwhile, junior can the right person for the job.)
4 hours means that I can do it within half a day on weekend and then spend the rest of the day with something else - going somewhere. At minimum, I can send kids outside with someone and be 4 hours alone at home to focus. Or I can do it 2 hours one evening and 2 hours another one - two days.
6 hours means that I spent most day with it. Expecting them to be outside 6 hours in row is a lot and so is blocking one room for that long. It would also mean 3 evenings not just 2.
> At the same time we are now looking for less senior engineers.
That is cool. (Really, all too many companies like to pretend that juniors don't exists or cant learn or just should magically become seniors instantly. Meanwhile, junior can the right person for the job.)
>4 hours means that I can do it within half a day on weekend and then spend the rest of the day with something else - going somewhere. At minimum, I can send kids outside with someone and be 4 hours alone at home to focus. Or I can do it 2 hours one evening and 2 hours another one - two days.
Makes sense, thanks for answering. Yeah for 6 hours you probably need to leave work a little bit early or work until late at night (if you want to fit it in your work week).
The way we setup the take home, you can't work on it for several evenings in a row : we have a 24h timer starting when you first see what you have to do. So you can do it over a full day if you want (like 2 hours in the morning and 2 more in the evening) but you can't work on it 60 hours over a week (that's the main potential issue I can see with take homes .. it encourages you to spend a lot of time in order to show off)
>Really, all too many companies like to pretend that juniors don't exists or cant learn .
Yeah, in all fairness a junior is a lot of work. You basically make the bet to lose 20% (made up number) of productivity for a year in order to mentor somebody in the hope that this person will stay for a while and grow.
However, we already spend so much time interviewing people that honestly I can't be convinced that only hiring 'best in the world' engineers is such a great idea.
Makes sense, thanks for answering. Yeah for 6 hours you probably need to leave work a little bit early or work until late at night (if you want to fit it in your work week).
The way we setup the take home, you can't work on it for several evenings in a row : we have a 24h timer starting when you first see what you have to do. So you can do it over a full day if you want (like 2 hours in the morning and 2 more in the evening) but you can't work on it 60 hours over a week (that's the main potential issue I can see with take homes .. it encourages you to spend a lot of time in order to show off)
>Really, all too many companies like to pretend that juniors don't exists or cant learn .
Yeah, in all fairness a junior is a lot of work. You basically make the bet to lose 20% (made up number) of productivity for a year in order to mentor somebody in the hope that this person will stay for a while and grow.
However, we already spend so much time interviewing people that honestly I can't be convinced that only hiring 'best in the world' engineers is such a great idea.
I'd bet few people will turn down an opportunity because of a couple hours spent working on a project. If they got this far in the interview process they should already know if it's a place where they wanna work. This is needless elitism. We hire tons of senior devs and small programming challenges are miles better then white-boarding.
>More senior and skilled people will decline to participate, because they know they can get a better return using the time otherwise spent on the coding challenge to network for more and easier opportunities.
Absolutely this. No top would put up with this unless the company is paying an ungodly sum of money.
Absolutely this. No top would put up with this unless the company is paying an ungodly sum of money.
I agree with you, maybe it's because I spent so much time consulting. My small project rate is $125/hr. And I've usually got someone lined up who is willing to pay me for whatever free time I can offer. The potential return on your free coding puzzle game would have to be pretty insane to get me to give up real guaranteed money.
A better option might be to just hire me as a consultant for some real ticket of yours and let me work it in off hours. Then we both win.
A better option might be to just hire me as a consultant for some real ticket of yours and let me work it in off hours. Then we both win.
I haven't gone through the traditional interview process since 1999, and that's just because I was only a few years in and wanted more money. Every job I've ever gotten since then has been from someone I've worked with before. They tell their boss, they take me out to lunch and it's a done deal.
There's a good chance somebody on the other end's going to run your program without really vetting it. Especially if you include a convenient "test runner." Not a bad attack vector, just get your alias past the first screen and you might get arbitrary execution inside their firewall on a host with access keys and the like.
I failed a take home challenge from Netflix and was a sad with how it went down. I was given 24 hours to complete the assignment and told that I 1.) shouldn't try to be clever, and 2.) that it shouldn't take more me more than 2 to 3 hours to complete. Early on I realized that the only way to get optimal performance was to use an interval tree to store my data, but thought that an interval tree was being "too clever". I implemented a more naive solution and submitted it with comments indicating that I believed that an interval tree was the optimal data structure to use in this case.
The response was that an interval tree was the correct data structure to use in order to pass the performance component of the assignment and that they were not interested in continuing with me any further. In hindsight I should have reached out to validate whether an interval tree was "too clever", and I think that was a more important factor in my failure than anything.
The response was that an interval tree was the correct data structure to use in order to pass the performance component of the assignment and that they were not interested in continuing with me any further. In hindsight I should have reached out to validate whether an interval tree was "too clever", and I think that was a more important factor in my failure than anything.
Once you are hired, will your coworkers at Netflix communicate better with you?
Wow, so sorry to hear this! This seems to test nothing other than what your interpretation of "clever" is. I've definitely learned that it never hurts to ask questions in these situations.
Man that makes me so upset for you
If you're thinking of interviewing your candidates with such methods, it's worth noting that their current employment agreement might prohibit them from participating due to IP clauses (more exactly, that they require permission to release any code that they've written, even in their spare time).
This can restrict your pool of candidates quite dramatically, as most US companies seem to have their engineers sign such agreements.
This can restrict your pool of candidates quite dramatically, as most US companies seem to have their engineers sign such agreements.
I was hired by my current employer following a take-home coding challenge. I consider the experience very positive, and it has allowed us to hire skilled developers. However, I notice a few differences between my experience and those of people who complain about these challenges:
1. The challenge was timed, and limited to 3 hours. This is critical, because it respects the applicant's time. 2. Similarly, other interviewing was limited to a very brief "are you actually a developer at all" phone screen and a 30 minute personality interview. 3. I was allowed to choose the framework and language I was most comfortable with, demonstrating that the employer was interested in how well I worked, not in how well I knew the particular framework chosen for their product. I think it probably helped that I used the same stack advertised in the job description, but I appreciate the flexibility being there.
Interviewing is a separate skill from software development. A take-home challenge isn't necessarily going to tell you everything about a person, but it seems to sort candidates into high/medium/low skill levels with a higher degree of accuracy than a paper or whiteboard tech interview.
1. The challenge was timed, and limited to 3 hours. This is critical, because it respects the applicant's time. 2. Similarly, other interviewing was limited to a very brief "are you actually a developer at all" phone screen and a 30 minute personality interview. 3. I was allowed to choose the framework and language I was most comfortable with, demonstrating that the employer was interested in how well I worked, not in how well I knew the particular framework chosen for their product. I think it probably helped that I used the same stack advertised in the job description, but I appreciate the flexibility being there.
Interviewing is a separate skill from software development. A take-home challenge isn't necessarily going to tell you everything about a person, but it seems to sort candidates into high/medium/low skill levels with a higher degree of accuracy than a paper or whiteboard tech interview.
Please don’t normalize “take-home coding challenges”.
At least make them instead of, rather than in addition to, whiteboard coding sessions.
How about neither. Should be: talk to the guy, talk about prior work. If you think they fit, hire them. If they end up being bad, fire them.
I've worked with and interviewed enough people who can talk a good talk and can't code their way out of a paper bag. I would rather figure that out before investing in onboarding someone.
This is going to sound worse than I intend it to, but If you can't tell when a programmer is blowing smoke up your ass during an interview, you are either asking the wrong questions or you need to have someone else in there with you.
Interviews, technical or not is a psychological process. Ask them about their previous projects, ask them what their favorite projects were and why. Ask them what their least favorite projects were and why. Ask them what the hardest thing they've worked on to date was and why.
Interviews, technical or not is a psychological process. Ask them about their previous projects, ask them what their favorite projects were and why. Ask them what their least favorite projects were and why. Ask them what the hardest thing they've worked on to date was and why.
I would have to agree with parent post though. There are always people who can talk they way through well but can't code. Its actually amazing to watch the real pros at this deflect all the tough questions. It is a great talent for something likes PR, sales or marketing but it might not be programming.
I've only hired one person like that and it was my fault. I was bedazzled by his color printouts of his UI. It was probably fine, but he was super lazy.
How many programmers have you hired using this method? How many have you fired?
I worked at a place once where they got so tired of trying to figure out which interview practices were best that they just started telling the recruiting firm to send people in for their first day of work without even bothering to interview them. This was a big local contact, so the recruiting firm wasn't going to send in known duds. The results worked out about the same as when we had actually done our traditional interview process.
I don't understand why people put up with all the bullshit that goes into tech hiring these days. I mean are companies that offer "market rates," putting people through this? No wonder they can't find anyone good.
So if take-homes are to become the best practice for interviewing, perhaps there should be some service for re-using old ones that companies can trust that the candidate did for some other company and under which constraints. Otherwise it's just too much of a time waste for candidates, repeating take-homes for every single candidate company.
It would be like personal projects but with a few guarantees on top, like that the candidate got the work assigned by someone else, and s/he did it under some time/other constraints. And there would perhaps be a public free-form feedback from the other comapny (no rating). So the next comapny has to judge the work itself without ability to pre-filter based on whatever rating the previous one would have given (to avoid bias, because the scoring would be non-objective/non-standardized).
But I would still not like the take-homes. :)
It would be like personal projects but with a few guarantees on top, like that the candidate got the work assigned by someone else, and s/he did it under some time/other constraints. And there would perhaps be a public free-form feedback from the other comapny (no rating). So the next comapny has to judge the work itself without ability to pre-filter based on whatever rating the previous one would have given (to avoid bias, because the scoring would be non-objective/non-standardized).
But I would still not like the take-homes. :)
I dunno, I’ve just walked from interviews that want take home tests. Plenty of other offers out there that don’t require it. If my
Github and past experience are not enough then we’re not a match. Companies are going to put you on a probationary period anyways so if they don’t like your work they’d fire you then.
Sure - as someone who transitioned into development from another industry, I could only rely on my past experience for transferrable skills, and not my actual technical skills. While Github is great, my Github is not always an accurate reflection of where I am technically as I continue learning.
Jane,
It's not a reflection on your or the article, it's more of a reflection of how silly tech interviews are these days.
It's not a reflection on your or the article, it's more of a reflection of how silly tech interviews are these days.
That might work for you, but there are a ton of applicants who are competent but do not have a great github profile. If you were hiring someone would you ignore all candidates who have an empty github? If so, what happens if you never find someone with a good profile? These are the problems we deal with when hiring.
Furthermore, for entry level positions -- candidates right out of college -- how do you measure competency?
Furthermore, for entry level positions -- candidates right out of college -- how do you measure competency?
I don’t think there’s any excuse why someone wouldn’t have something to show that they’ve built on their own. If they have nothing and no past experience I don’t know why’d they’d even make it to an interview.
If there is a guide or a book for taking an interview, the interview has already failed and is measuring the wrong thing.
I disagree. Even if you had a hypothetical perfect interview that tested exactly the right combination of raw logical ability/concept familiarity/effective communication, there would always be room for a guide that reminded people exactly how to best prepare.
Interviews need to test for certain characteristics in a limited amount of time. Even if there was some way to say "here's the code I've wrote, e-mails I've sent, and videos of me leading team meetings in the last year", no company would have the time to look at those materials.
As such, we're stuck with the common question formats trying to determine if you communicate with your interviewer well (understanding what's being asked, explaining your thought process and what you're trying to accomplish, talking about anecdotes and building rapport) while also showing off some technical understanding of implementation and design. No one's yet found a better system that's more respectful of both the company's time and the candidate's time.
These "take home projects" are an interesting variation that I'm not going to dismiss out of hand, but it's hard enough to take 8 hours out of my day for an interview loop - adding more hours in the evenings isn't something that scales well to interviewing with a bunch of places.
Interviews need to test for certain characteristics in a limited amount of time. Even if there was some way to say "here's the code I've wrote, e-mails I've sent, and videos of me leading team meetings in the last year", no company would have the time to look at those materials.
As such, we're stuck with the common question formats trying to determine if you communicate with your interviewer well (understanding what's being asked, explaining your thought process and what you're trying to accomplish, talking about anecdotes and building rapport) while also showing off some technical understanding of implementation and design. No one's yet found a better system that's more respectful of both the company's time and the candidate's time.
These "take home projects" are an interesting variation that I'm not going to dismiss out of hand, but it's hard enough to take 8 hours out of my day for an interview loop - adding more hours in the evenings isn't something that scales well to interviewing with a bunch of places.
don't do unpaid take-home coding challenges. particularly if you have no idea what the company is offering for compensation should you get the offer.
For a well established developer, this may be true, but for newer developers, they often have to jump through a lot of hoops to prove their skills first, especially if they come from a non-traditional background and do not have a CS degree.
Do you think this issue is less pronounced with the move toward companies providing compensation guidelines up front?
Do you think this issue is less pronounced with the move toward companies providing compensation guidelines up front?
I only agree so far as the take-home challenge has no time restrictions.
I would choose a take-home coding challenge over any white boarding challenges.
I would choose a take-home coding challenge over any white boarding challenges.
Agree
Agree
Disagree
I think it's awesome that you can become a software engineer without any engineering schooling. Where I live you have to take software engineering in University to be a software engineer. Bootcamps and certs usually only get you dev jobs.
Many companies use the terms "software engineer" and "developer" and "programmer" and "analyst" interchangeably. There's no universal agreement on the difference. I'm not saying it should be this way, but it is.
In my market, a title of software engineer generally means higher compensation.
Even though most of us are not real engineers who studied BEs, the title has more prestige than developer (or god forbid, programmer).
Even though most of us are not real engineers who studied BEs, the title has more prestige than developer (or god forbid, programmer).
It’s a title arms race and that’s how it’s going to be until it’s illegal to call yourself an engineer w/o an engineering degree, such as in Texas and Canada.
They're all meaningless.
Instead of take home challenges, I think I would rather give a potential employee some code snippet and ask them to explain to me in their own way as to what is going on and if they see/find any particular issues.
This will theoretically let me gauge their attention to detail. Ideally I think it would show me how familiar they are with the programming language the snippet was written in as well.
This will theoretically let me gauge their attention to detail. Ideally I think it would show me how familiar they are with the programming language the snippet was written in as well.
Flagged. Since no payment is mentioned, so it is assumed that a candidate will do it for free... Lets not encourage such deeds.
Some people spend an entire week on the coding challenge, which really set the bar unrealistically high. Once I turned in a project 2 hours after receiving the assignment and got rejected because they think I didn't even try. Every functionality worked flawlessly, but they want tests and for me to treat it like a real production work. But how do you motivate yourself to treat such a trivial take-home (which will never be maintained because its a take-home assignment) with the perfection you would in production code? In production you have much better insights like how your component impact others, how users will be using your app, etc.