Becoming a dungeon master for an interview(propelauth.com)
propelauth.com
Becoming a dungeon master for an interview
https://www.propelauth.com/post/interview-dungeon-master
180 comments
I like to think it went like this
1. Interviewer: If you're a good software engineer, you can answer basic algorithmic questions.
2. Interviewees: Practice algorithmic questions so you appear to be a good software engineer.
3. Interviewer: People are just studying leetcode to get jobs, what can we do? Ask harder leetcode questions.
4. Other companies: Let's copy them since they're successful.
In short, the questions used to be reasonable until people specifically prepared for them. No one knew what to do about it so they just raised the difficulty, which made it even more unfair for people who don't specifically prep.
1. Interviewer: If you're a good software engineer, you can answer basic algorithmic questions.
2. Interviewees: Practice algorithmic questions so you appear to be a good software engineer.
3. Interviewer: People are just studying leetcode to get jobs, what can we do? Ask harder leetcode questions.
4. Other companies: Let's copy them since they're successful.
In short, the questions used to be reasonable until people specifically prepared for them. No one knew what to do about it so they just raised the difficulty, which made it even more unfair for people who don't specifically prep.
FizzBuzz as an example of 2/3: people used to occasionally talk about how interviewees were just memorizing the answer, and when they tweaked it slightly (like adding a 3rd number), a bunch of them could no longer solve it.
“if a flag of truth were raised we could watch every liar rise to wave it”
I heard this lyric at a formative time, and I’ve seen it be proven true many times. Including tech interviews. People continually seek out those signals that imply knowledge and experience and even shared culture, but those signals inevitably become too small (smaller = quicker and easier to weed people out) and then they become the very things that people practice in order to look like they have the knowledge, experience, or shared culture they need in order to get through the doors and secure the opportunities.
Then those signals get burned and the cycle starts again (in fact, in my experience the cycles concurrently overlap).
I heard this lyric at a formative time, and I’ve seen it be proven true many times. Including tech interviews. People continually seek out those signals that imply knowledge and experience and even shared culture, but those signals inevitably become too small (smaller = quicker and easier to weed people out) and then they become the very things that people practice in order to look like they have the knowledge, experience, or shared culture they need in order to get through the doors and secure the opportunities.
Then those signals get burned and the cycle starts again (in fact, in my experience the cycles concurrently overlap).
It's actually great on the hiring side — you can skip all that bullshit and because tryhards are all prepping obscure CS questions just having a conversation about technical topics has become signal again. Measure something people aren't trying to game and you get a better assessment, go figure.
Yes, I explicitly do the opposite and have the most pragmatic exercises, questions.
Another thing is an exercise for system design where hyperscaling is not required and the thing is actually quite simple. Many who have specifically prepared by leetcoding and reading "cracking the coding interview" 10 times over will naturally overengineer everything trying to fit this exercise to those book patterns dropping all common sense, all the while not having actually built anything meaningful.
I think these people will mostly try to rest and vest anyway. Truly passionate people will pass since they have actually built something and will understand the exercise.
Another thing is an exercise for system design where hyperscaling is not required and the thing is actually quite simple. Many who have specifically prepared by leetcoding and reading "cracking the coding interview" 10 times over will naturally overengineer everything trying to fit this exercise to those book patterns dropping all common sense, all the while not having actually built anything meaningful.
I think these people will mostly try to rest and vest anyway. Truly passionate people will pass since they have actually built something and will understand the exercise.
When I get a system design question I always tell the interviewer "I'd just run it on a single server with an SQLite backend, that will be plenty for the median software service and you haven't told me any numbers that suggests this needs more" and then it turns out they wanted it to run at the scale of WhatsApp.
For what it's worth if I ever had a candidate give me this answer on a systems design problem I'd probably immediately stop evaluating them and start selling them on the role.
Usually it is expected that you ask yourself what is the expected traffic, but I usually answer with a number that technically one box could handle.
"So what scale are we talking about? A few million monthly users? So like hackernews? I would use a single server... "
You really need two in physically different locations (ideally ASes) with some form of failover, assuming you want a reasonable guaranteed uptime
Does 99.995% [1] of hackernews sound reasonable enough to you?
The reality is a lot of systems (especially simple ones) run perfectly fine on a single server with next to no downtime and all the additional redundancies we introduce also add additional points of failures and without the scale that makes these necessary you might actually end up reducing your availability.
[1] https://hn.hund.io/
The reality is a lot of systems (especially simple ones) run perfectly fine on a single server with next to no downtime and all the additional redundancies we introduce also add additional points of failures and without the scale that makes these necessary you might actually end up reducing your availability.
[1] https://hn.hund.io/
> Does 99.995% [1] of hackernews sound reasonable enough to you?
How did you come up with that number? I looked at the link and just one of the outages listed on January 10 was 59 minutes. That alone makes the uptime worse than 99.99% for the entire year before it was halfway through January.
(99.995% means at most 26.3 minutes of downtime per year. See https://en.wikipedia.org/wiki/High_availability#Percentage_c...)
How did you come up with that number? I looked at the link and just one of the outages listed on January 10 was 59 minutes. That alone makes the uptime worse than 99.99% for the entire year before it was halfway through January.
(99.995% means at most 26.3 minutes of downtime per year. See https://en.wikipedia.org/wiki/High_availability#Percentage_c...)
it's the 30 day uptime statistic displayed on the website I linked.
if you click the history it says 98.452% over 365 days.
if you click the history it says 98.452% over 365 days.
That’s a pretty low level. That’s lower than my pi hole. But I wouldn’t consider my pihole to be anything other than best endeavours (99.1%). Two would be fine, but there are common points of failure which would limit the solution
I can't see how hackernews is 99.995% if I get at least 30-40 "Can't serve your request" error pages a year.
HN is on a pair of servers.
The second one is just a standby though and not in another region. And if I recall correctly dang mentioned at an outage that the failover to the standby is manual. But I'm not sure.
https://news.ycombinator.com/item?id=16076041
https://news.ycombinator.com/item?id=16076041
This so much this! Experience is such a wonderful thing.
We've run 375,000 self-service employees ERP system (so much heavier single transaction than HN) on a single (large:) db2/aix box with no downtime over last 8 years. That's well within published specs for that hardware / software combo.
Yes we do have a DR box in another data centre now, in case of meteorite or fire.
This used to be the norm. A single hardware / software box CAN be engineered for high uptime. It's perfectly fine when we choose to go the other way and distribute software over number of cheap boxes with meh software, but I get pet-peeved when people assume that's the ONLY way :).
Yes we do have a DR box in another data centre now, in case of meteorite or fire.
This used to be the norm. A single hardware / software box CAN be engineered for high uptime. It's perfectly fine when we choose to go the other way and distribute software over number of cheap boxes with meh software, but I get pet-peeved when people assume that's the ONLY way :).
I’m paranoid about environmental situations so always have a failover, but then my DR plans include events such as “Thames barrier fails”.
Add a static caching layer and you’re ready for traffic spikes.
The highest correlation I see to success in my field is a background of PC gaming. They tend to do better on interviews in the technical regard, all the flashy certificates go out the window of you cant tell me what you would do if a computer wont post.
absolubtely! One of the best jobs I ever worked at asked me what CPU I had in my computer at home, I talked about how i'd built my own pc and the parts that went into it and why, and they (much) later when I was settled at the role said they could see I was a good candidate from that point onward. I think its useful to show a curiosity about your tools, about the boundaries of your world, which is useful when things arent going to plan
Do you really think it made you a better candidate, or did this person just feel a connection due to a similarity in interests or in your approach to computing? Was it even relevant for the job?
I find basic pc troubleshooting skills to be highly relevant to working in a data center. I learned about isolation testing and minimum config when I was like 13 trying to play MMOs, these things come relatively naturally to me due to exposure at a young age. That interest has prevailed well into my adulthood now, having no formal education I run circles around the relatively disinterested comptia kids.
As to being a better candidate, there is little I doubt less* than simple observations I can make at work. We could split hairs over causality, but there is a clear distinction between the people who go home to delid their CPU and the people who have devoted their time to certs instead.
Of course there will always be those sages that dont really care much for videogames and have transcended the street knowledge, those people 1. Have better jobs to work 2. Are harder to find with little benefit.
As to being a better candidate, there is little I doubt less* than simple observations I can make at work. We could split hairs over causality, but there is a clear distinction between the people who go home to delid their CPU and the people who have devoted their time to certs instead.
Of course there will always be those sages that dont really care much for videogames and have transcended the street knowledge, those people 1. Have better jobs to work 2. Are harder to find with little benefit.
> due to a similarity in interests or in your approach to computing?
It's hard to say for sure - both of these things are also part of being a good candidate and working well in a team.
But I do think that this experience is both because of and forms part of who I am, a certain troubleshooting & optimizing mindset and curiosity with machines. Is it strictly necessary, or will all people with this shared hobby/past be this way? I don't know. I do think its a useful or good fuzzy signal, to be best used alongside other signals.
Was it relevant for the job? Not for the job description. There were times when I used related experience to solve problems or smooth things over though, such as figuring out why a QA engineer's setup was bluescreening (faulty ram), or in having familiarity with tools built into windows for performance profiling and debugging memory & storage problems with programs
It's hard to say for sure - both of these things are also part of being a good candidate and working well in a team.
But I do think that this experience is both because of and forms part of who I am, a certain troubleshooting & optimizing mindset and curiosity with machines. Is it strictly necessary, or will all people with this shared hobby/past be this way? I don't know. I do think its a useful or good fuzzy signal, to be best used alongside other signals.
Was it relevant for the job? Not for the job description. There were times when I used related experience to solve problems or smooth things over though, such as figuring out why a QA engineer's setup was bluescreening (faulty ram), or in having familiarity with tools built into windows for performance profiling and debugging memory & storage problems with programs
Sadly, we can’t defer to stack overflow for interview success like we can with code. GPT will/may help break it up but until we stop with the sociology questions and get back to technical delivery, we’ll continue to see people try to game a system and then try to game that system they gamed. It’s real life NPM.
What changed? Well it was about 2010, The industry became a target for very low quality people to do whatever they could to land the job by faking and lying. All of these annoying interview tricks are just a way to try to filter out all the people (90% of them now) who taught themselves how to talk the talk but couldn’t program themselves out of a wet paper bag. And yes, there are ways to figure it out but it’s really hard given the current volume, and having your engineers stop and interview constantly kills productivity when you have to interview 20 people and only one can even count words in a file.
Function wetpaperbag() {
Quit()
}
How did I do?
How did I do?
Not very well, in all honesty.
The function name is pretty misleading, since it looks like a constructor. Even if it's not a constructor, the functionality is pretty far away from the name you've chosen, so it's not a compelling API design either way. How would others use this function? How is this name self-documenting? The decision here is lacking consideration on all fronts.
Also, while I don't immediately recognize the exact language, the style looks off too. The function wetpaperbag is declared in all lowercase, but inside it's calling a function Quit that's capitalized? And, whether this particular language is camel-casing or snake-casing, it's clear that "wetpaperbag" isn't going to pass linting either way.
Overall, strong no hire. Next.
(/s... but only mostly, since the GP's point is that you actually can get a lot out of just asking simple interview questions, which I think we've helpfully proven out here)
The function name is pretty misleading, since it looks like a constructor. Even if it's not a constructor, the functionality is pretty far away from the name you've chosen, so it's not a compelling API design either way. How would others use this function? How is this name self-documenting? The decision here is lacking consideration on all fronts.
Also, while I don't immediately recognize the exact language, the style looks off too. The function wetpaperbag is declared in all lowercase, but inside it's calling a function Quit that's capitalized? And, whether this particular language is camel-casing or snake-casing, it's clear that "wetpaperbag" isn't going to pass linting either way.
Overall, strong no hire. Next.
(/s... but only mostly, since the GP's point is that you actually can get a lot out of just asking simple interview questions, which I think we've helpfully proven out here)
The people who are fooled by fast talkers don’t care about the results of leetcode questions either.
Even just giving Leetcode questions isn’t enough anymore…
I’ve had plenty of people just memorize the top 50 used questions and be able to answer them, but then not be able to explain anything about their answer. It’s exhausting.
Explaining the reasoning while solving the code puzzle is the most important part, though. Just being able to solve them without speaking a word has never been enough. Unless you are talking about automated coding tests (but that also has never been enough).
It's super easy to filter for that though. Just change the problem slightly. If they give the rote answer you know what you need to know. It does require that you know the answer though. :)
> I'm curious what changed around 2015-2016 that led to the current interview process. In the before times the interview was more of a technical conversation, sure they had some gotcha questions, but nothing too brutal. If you had real experience that you co...
Places just copied big tech companies that were already doing that for many years prior to 2015. I had leetcode style interviews well before 2010 even.
Places just copied big tech companies that were already doing that for many years prior to 2015. I had leetcode style interviews well before 2010 even.
Part of the problem is people just memorize things for interviews. We interviewed three candidates that were almost exact copies of each other in their responses. Maybe they all read the same interview book?
I also witnessed someone give a completely wrong (algorithm) whiteboard answer to a problem. Obviously they weren't thinking about the problem as they did it.
I also witnessed someone give a completely wrong (algorithm) whiteboard answer to a problem. Obviously they weren't thinking about the problem as they did it.
Algorithm that were solved over decades and centuries are asked during the 15-20 minute demonstration asks it to be memorized
The problem is the sheer dumbness of those who believe it is important in the interview process
The problem is the sheer dumbness of those who believe it is important in the interview process
Plus, everyone thinks they are Google or Facebook, or that they are about to become Google or Facebook. You are not.
Pretty much this. Companies needed software engineers, looked at what Google was doing, and copied. And they call themselves innovative for doing it.
Ah, so to reach their hiring destination, their preferred mode of transportation is the bandwagon.
Methinks it’s a good enough method to somehow get good enough candidates. Hiring is hard because it’s a negotiation: sellers always upsell their skills and buyers gotta figure it out. Negotiation is skill that not everyone has so at least by having some sort of standard test like the current leetcode system, you can guarantee the dudes coming in know the difference between a Stack and Queue
A company whose smart engineers threw their hands up claiming no moat while they were the pioneer of AI
After subprime interest rates went to or near zero, prompting a huge influx of cash available for VCs with heightened FOMO due to increasing complexity of software. Marginal quality of talent decreased while remuneration went through the roof, drowning out valid signals in cvs and interview processes with so much noise that any complicated enough system was seen as valid. Extremely high rate of failure, which was made acceptable due to the potential rewards of funding a unicorn early further degraded the ability of anyone involved to conduct clear assessments.
Big Tech could afford to do that due to their aura of mysticism and greatness, but if you're a mom and pop shop with Google-esque aspirations but none of the compensation and status, I wonder how many candidates have the patience and interest to go through the hoops.
Before 2010, I used to get a lot of “why are manhole covers round?” type questions, not ACM competition type questions.
It all started in a few years earlier. Google had determined that there was a strong correlation between understanding fundamental computer science and being a strong software engineer, so they began testing for computer science fundamentals. At the time lots of companies were trying to mimic Google's success and it quickly became an industry standard. People began just studying how to answer these questions. The population of people that are good at these types of interviews shifted from hardcore detail oriented programmers with academic backgrounds to mostly people that spend time practicing leetcode style questions.
One of their hiring criteria is cognitive ability. If it correlates with strong proficiency in CS algorithms, then this might have been why they focus on it when hiring engineers.
(I am not saying that this strong correlation exists.)
(I am not saying that this strong correlation exists.)
The correlation doesn't exist. A large segment of the Leetcode-uber-alles practitioners are also the ones that are obsessed with total compensation as the ultimate measure. It was a mistake by google and others to focus purely on this , but we are where we are. Many of the leetcode fanatics can't deal with real world issues in software or operations.
These companies look for people who write software to handle real-world issues on software and operations.
The mistake is in refusing to have a separate job and hiring rubric for sysadmins
The mistake is in refusing to have a separate job and hiring rubric for sysadmins
[deleted]
I'm grateful for leetcode. Despite my background in electrical engineering, where I specialize in statistical signal processing, I never had the opportunity to delve into algorithm or data structure courses during my college years.
In my field, understanding signals and systems, probability, optimization, and numerical computing is very important.
Leetcode, in comparison, offers a more confined scope, making preparation more manageable and systematic, ultimately helping me break into software engineering.
In my field, understanding signals and systems, probability, optimization, and numerical computing is very important.
Leetcode, in comparison, offers a more confined scope, making preparation more manageable and systematic, ultimately helping me break into software engineering.
We might also ask why EE specialists (and indeed half of STEM) needs to fall back to software engineering to get meaningful and stable employment, but capitalism probably does not like us asking these pesky questions.
It's not that EE *needs" to fall back on software for stable employment, it's that software pays more, and ridiculously more at big tech companies.
I have both EE and CS degrees, but I make literally more than double in CS-type jobs than I would back in EE. Even though back in the day, I was definitely better at EE stuff than CS stuff.
This was true very early career, and for now at least continues to be true.
---
I'm still of the opinion that software will eventually revert to the mean, comparable to other engineering compensation... some year. It probably won't happen until there's significant regulatory and liability burden that doesn't really exist in software yet. Like some in software complain about GDPR or DMA or whatever, but crank that up orders of magnitude and then add career-and-company-ending lawsuit threats on top, and then you have real engineering. Then it becomes much more expensive to do things, profit margins go way down, and pay does as well.
Who knows when that'll actually happen. For now, software is bonkers profitable.
(When software engineering becomes real engineering, I posit that it will also cease to be as highly paid)
I have both EE and CS degrees, but I make literally more than double in CS-type jobs than I would back in EE. Even though back in the day, I was definitely better at EE stuff than CS stuff.
This was true very early career, and for now at least continues to be true.
---
I'm still of the opinion that software will eventually revert to the mean, comparable to other engineering compensation... some year. It probably won't happen until there's significant regulatory and liability burden that doesn't really exist in software yet. Like some in software complain about GDPR or DMA or whatever, but crank that up orders of magnitude and then add career-and-company-ending lawsuit threats on top, and then you have real engineering. Then it becomes much more expensive to do things, profit margins go way down, and pay does as well.
Who knows when that'll actually happen. For now, software is bonkers profitable.
(When software engineering becomes real engineering, I posit that it will also cease to be as highly paid)
I love hardware and EE, but most of my friends that do it hate it and work on power lines only. I wish hardware was as good, but it's not as in demand and it's harder too!
"capitalism" Is it in the room with you? Do you need assistance? Did the evil capitalism touch you in your naughty place?
But seriously... everything uses computers and programming ties everything together. I don't think you have to be a grand master programmer but having those skills make you a better employee.
The real pesky question is why do you think socialism, communism or the alternatives to "capitalism" wouldn't benefit from having people able to use better tools? Productivity multipliers would help "The People" and The Masters in charge of the authoritarian dictatorship alternatives to "capitalism"...
But seriously... everything uses computers and programming ties everything together. I don't think you have to be a grand master programmer but having those skills make you a better employee.
The real pesky question is why do you think socialism, communism or the alternatives to "capitalism" wouldn't benefit from having people able to use better tools? Productivity multipliers would help "The People" and The Masters in charge of the authoritarian dictatorship alternatives to "capitalism"...
If you don't have the background, to me you would be a risky bet since you may be missing some fundamentals. But I'm not sure what niche you're specializing in, or if you could provide some social proof that you've otherwise picked up the requisite software engineering skills.
If sounds like gatekeeping, it kind of is.
If sounds like gatekeeping, it kind of is.
I recently started interviewing again and haven't encountered many leetcode style questions this time around. I'm noticing lots of contrived questions that relate to some feature the company had to build.
What does it mean for an interview question to be contrived?
So overly specific that it feels inappropriately artificial within the context of testing a stranger.
Yeah, the kind of question that I'm not comfortable giving a vague nonanswer to because I'm not familiar with all the intricacies they spent months ironing out. Which then turns into "what do you want me to say" whackamole of endless questions to gain the context needed to actually answer their question. Usually asked by people who cannot provide the context that actually led to the answer but still believe they understand why it was chosen.
It's a pretty specific scenario but it's happened to me
It's a pretty specific scenario but it's happened to me
If the question is sufficiently related to the actual application, it could be an interview of the "solve our biggest problems for us, unpaid, without context" form, which I experienced once at a tiny startup. The technical lead just sat there silently as I probed with more questions. Eventually, I set down a boundary and let them know that that is something for the business to solve.
Maybe contrived wasn’t the best word, but I’ve been asked to implement a simplified feature of the company’s product. So at one company I was asked to create a simplified model for their product with support for undo/redo.
This is my favorite, “we want you to work for free”
No they definitely aren't using my code in any way. During the interview they are trying to see how I approach building something they have already completed.
I just wish that if interviewing was going to become “random irrelevant stuff from second or third year CS” that we had gotten automata theory, because that was actually fun.
I recall being on a software engineer interview loop (faang), my topic to interview for was around practical problem solving. The candidate performed poorly, unable to write any solution to a basic log parsing problem in the language if their choice. In the debrief, apparently the candidate did very well in the algorithmic interview part of the loop. I dissented on the decision to hire, because of their inability to solve basic real world problems, the group proceeded anyway..
And this Dungeon Master tool can add some ambience to your adventure)) https://tabletopy.com
I had an interview recently where I was given a 1000 line, deeply stacked conditional and asked to traverse it. Like, what are they selecting for? I hope that’s not demonstrative of their source code.
You could always learn the solutions once and put them into a set of Anki flashcards, so that you occasionally get reminded to practice them just before you forget them. You really don't have to keep repeating the 3 months "from scratch" experience. That's what I have done, pretty badly to be frank, and the prep time is now closer to 2 weeks.
Compared to many other industries where people are required to go get expensive and time consuming "continuing education", I don't think keeping a timed flashcard deck of maybe a couple hundred items total in rotation is all that onerous.
Compared to many other industries where people are required to go get expensive and time consuming "continuing education", I don't think keeping a timed flashcard deck of maybe a couple hundred items total in rotation is all that onerous.
Leetcode started around 2015
Personally I love leetcode. I do it on Sundays like others do crossword puzzles.
But once upon a time these leetcode interviews made a ton of sense. Just getting your program to run at all was basically a leetcode exercise.
see for example: https://arstechnica.com/gaming/2020/03/war-stories-how-princ...
I have my war stories too. It was fun but frustrating. Definitely not as productive as today.
Early 2000s internet was kind of like that too. The web was huge relative to what we had to work with and there wasn't the kind of products/services available then so asking leetcode stuff had a logic to it.
Admittedly, that's not where we are now for most programmers so it makes sense that the interview process should adapt.
But once upon a time these leetcode interviews made a ton of sense. Just getting your program to run at all was basically a leetcode exercise.
see for example: https://arstechnica.com/gaming/2020/03/war-stories-how-princ...
I have my war stories too. It was fun but frustrating. Definitely not as productive as today.
Early 2000s internet was kind of like that too. The web was huge relative to what we had to work with and there wasn't the kind of products/services available then so asking leetcode stuff had a logic to it.
Admittedly, that's not where we are now for most programmers so it makes sense that the interview process should adapt.
The kind of people who do best at FAANG interviews are the ones who enjoy leetcoding. We’re all selecting for the wrong skills.
Even if I enjoy leetcoding and I do, there are tons of things I would rather do before that, and that are far more useful, such as building and working on side projects.
So to me leetcoding is a waste of time and less fun than actually building something, having other people use what you have built. And there is infinite things to build there and learn.
So to me leetcoding is a waste of time and less fun than actually building something, having other people use what you have built. And there is infinite things to build there and learn.
I get that. I only do medium and easy problems for a reason. Hard problems are hard and I want to solve it over morning coffee. :)
Then again, I'm sure all of my leisure activities are a waste of time on someone's metric.
I probably learn more from the comments than the problems though. People post lots of interesting idioms and it's interesting to me to see how others codify various standard algorithms. Especially across languages.
Then again, I'm sure all of my leisure activities are a waste of time on someone's metric.
I probably learn more from the comments than the problems though. People post lots of interesting idioms and it's interesting to me to see how others codify various standard algorithms. Especially across languages.
> Interviews should be as close as possible to the work that the candidate would do if they join.
Interviews should give information to both parties about what it would be like if the person joined.
Simulating a typical work task, as this article describes, might give you some of that. Though important to remember that's far from all that's relevant to either party.
A recently fashionable Leetcode hazing, on the other hand, is mostly "anti-pattern". It tells the company whether or not the individual has studied for that ritual specifically, and was willing to submit to it for the approval of this company (or is using this company as practice for the gameable ritual). And it tells the individual nothing except that the company thinks this is a good way to do interviews.
Interviews should give information to both parties about what it would be like if the person joined.
Simulating a typical work task, as this article describes, might give you some of that. Though important to remember that's far from all that's relevant to either party.
A recently fashionable Leetcode hazing, on the other hand, is mostly "anti-pattern". It tells the company whether or not the individual has studied for that ritual specifically, and was willing to submit to it for the approval of this company (or is using this company as practice for the gameable ritual). And it tells the individual nothing except that the company thinks this is a good way to do interviews.
I feel like leetcode is a Goodhart's Law thing. When nobody was cramming for it, then at some places it was a good measure.
Coding is more like weight lifting sometimes then running. There are some weights that some people just can't pick up. And so if the job involves lifting things that are usually light but occasionally heavy, then a squat becomes a good measure for candidate.
But once everyone knows a big squat is the way in, they train squats, they show up in lift suits, they use amyl nitrates and the whole thing stops being a good measure.
Because on the job you don't squat in a squat rack, but pick things up where they are found, in the conditions on the ground.
Goodheart's law breaks things, because now you have a bunch of people optimized for a task that isn't exactly the job.
Coding is more like weight lifting sometimes then running. There are some weights that some people just can't pick up. And so if the job involves lifting things that are usually light but occasionally heavy, then a squat becomes a good measure for candidate.
But once everyone knows a big squat is the way in, they train squats, they show up in lift suits, they use amyl nitrates and the whole thing stops being a good measure.
Because on the job you don't squat in a squat rack, but pick things up where they are found, in the conditions on the ground.
Goodheart's law breaks things, because now you have a bunch of people optimized for a task that isn't exactly the job.
> Goodheart's law breaks things, because now you have a bunch of people optimized for a task that isn't exactly the job.
Companies lose another way too.
Folks who might be darn good at picking things up decide it isn't worth the focused training to get into companies that only seem to care about squatting, so they don't apply.
Companies lose another way too.
Folks who might be darn good at picking things up decide it isn't worth the focused training to get into companies that only seem to care about squatting, so they don't apply.
Yeah, I feel like any measure like this starts as a way to measure a proxy for IQ. But as it becomes a known thing, people train for it and it becomes a measure of conscientiousness (In the big five sense, aka diligent trainers rise to the top).
> they use amyl nitrates
I think you meant ammonia. Poppers would cause an acute decrease in blood pressure which would be counterproductive during the lifting phase with a heavy weight.
I think you meant ammonia. Poppers would cause an acute decrease in blood pressure which would be counterproductive during the lifting phase with a heavy weight.
> Interviews should give information to both parties about what it would be like if the person joined.
I have participated in interview processes where, at the last interview, the employer tries to talk the candidate out of taking the job. Discuss the problems, the warts, the intractable issues, and other things that are downsides of this job. Every organization has these.
I love this, because the employee is going to find all of the issues out anyway, and it's better for both parties if they part before employment begins if the issues are too much.
I have participated in interview processes where, at the last interview, the employer tries to talk the candidate out of taking the job. Discuss the problems, the warts, the intractable issues, and other things that are downsides of this job. Every organization has these.
I love this, because the employee is going to find all of the issues out anyway, and it's better for both parties if they part before employment begins if the issues are too much.
Interesting, and I could see that working well for some places.
I do something related to this in the interview. Part of the interview is more like things you'd tell a colleague who was considering that: pretty accurate characterization of what you know of the work, your idea of pros, things that might or might not be cons for the person, etc.
Basically, they show up the first day, and their interactions with me are just the same (plus a little extra excitement on my part that they joined, and now able to share proprietary details but without unpleasant surprises). Six months later, and I'm still just the same (plus extra familiarity and increased mutual trust). And hopefully they also are what they seemed to be in the interview.
Though you can't necessarily be 100% candid about everything in an interview, like if you were doing an internal assessment a technology, product, vendor, or competitor. For example, there are personnel issues, company image, proprietary tech, etc. Also, you don't want to say something that ends up misleadingly quoted out of context on a jobs forum or word-of-mouth by someone with a different sense of professionalism or confidentiality, and you don't want to give ammo to a spy from a competitor.
In practice, the line of what to say seems pretty intuitive to hit, and I haven't knowingly had a problem with it. Though, when working at one company doing exotic-sounding, high-value-sounding stuff, I did have a couple candidates grill me on the tech and/or on resources. On those I try to respond something like, "Some of that is very proprietary, and unfortunately can't be talked about much right now until someone joins the team. But I can tell you some of the public information from news articles and such is A, B, and C. There's a few marketecture diagrams on our Web site, and links to some published research papers that detail those aspects better than I could."
I do something related to this in the interview. Part of the interview is more like things you'd tell a colleague who was considering that: pretty accurate characterization of what you know of the work, your idea of pros, things that might or might not be cons for the person, etc.
Basically, they show up the first day, and their interactions with me are just the same (plus a little extra excitement on my part that they joined, and now able to share proprietary details but without unpleasant surprises). Six months later, and I'm still just the same (plus extra familiarity and increased mutual trust). And hopefully they also are what they seemed to be in the interview.
Though you can't necessarily be 100% candid about everything in an interview, like if you were doing an internal assessment a technology, product, vendor, or competitor. For example, there are personnel issues, company image, proprietary tech, etc. Also, you don't want to say something that ends up misleadingly quoted out of context on a jobs forum or word-of-mouth by someone with a different sense of professionalism or confidentiality, and you don't want to give ammo to a spy from a competitor.
In practice, the line of what to say seems pretty intuitive to hit, and I haven't knowingly had a problem with it. Though, when working at one company doing exotic-sounding, high-value-sounding stuff, I did have a couple candidates grill me on the tech and/or on resources. On those I try to respond something like, "Some of that is very proprietary, and unfortunately can't be talked about much right now until someone joins the team. But I can tell you some of the public information from news articles and such is A, B, and C. There's a few marketecture diagrams on our Web site, and links to some published research papers that detail those aspects better than I could."
[deleted]
> was willing to submit to it for the approval of this company
I abstain, myself. I have too much real work that I'd like to push forward to be able justify spending weeks of free time on fake work.
I abstain, myself. I have too much real work that I'd like to push forward to be able justify spending weeks of free time on fake work.
Great! Isn’t that the point? That some people who really don’t want the job won’t invest a bit of their time and people who do really want the job will?
I doubt it?
My willingness to practice leetcode isn't correlated with my interest in new work or the work any given job post describes. It's also not correlated with my willingness to invest time in exploring the position.
6 hours of leetcode practice teach me nothing about a position, the work it entails, the problems the company is grappling with, and the people I'd be in the trenches with. 6 hours in conversation or working close to real problems are higher leverage.
I have meaningful open source work with real users in the queue, and blog posts that won't write themselves. I feel like it's pretty reasonable to sort positions that don't compel low-leverage busywork that doesn't help me assess the position above those that do.
Sure, they might mean to exclude me, but I'm skeptical.
My willingness to practice leetcode isn't correlated with my interest in new work or the work any given job post describes. It's also not correlated with my willingness to invest time in exploring the position.
6 hours of leetcode practice teach me nothing about a position, the work it entails, the problems the company is grappling with, and the people I'd be in the trenches with. 6 hours in conversation or working close to real problems are higher leverage.
I have meaningful open source work with real users in the queue, and blog posts that won't write themselves. I feel like it's pretty reasonable to sort positions that don't compel low-leverage busywork that doesn't help me assess the position above those that do.
Sure, they might mean to exclude me, but I'm skeptical.
I also think that might be true. Adding friction filters out candidates who have more choices, which are likely to be the top performers. Hardly anybody wants to work specifically for one particular company, so the people willing to continue despite friction may be more likely to be middle performers.
> My willingness to practice leetcode isn't correlated with my interest in new work or the work any given job post describes.
The job requires you to practice leetcode in order to get the job. If you don’t want to practice leetcode then you don’t want the job.
The job requires you to practice leetcode in order to get the job. If you don’t want to practice leetcode then you don’t want the job.
Since you're apparently here to play semantics games: if you aren't willing to pay your employer for the opportunity to solve their problems, you don't want the job.
Yes! You’re getting it.
“Paying your employer to work for them” is commonly referred to as a “scam”.
Studying and practicing to pass an interview isnt “paying your employer.”
That selects for people who aren’t holding productive jobs currently.
It’s a reasonable filter for junior positions, but seems counterproductive if you’re looking to fill a position where experience would be valuable.
It’s a reasonable filter for junior positions, but seems counterproductive if you’re looking to fill a position where experience would be valuable.
Don't forget about the people who don't have to spend weeks or months focused just on interview prep to do well on the average leetcode question...
That's who interviewers are looking for. Now, a case can be made that there aren't that many of those people and you might not need them anyway, but... if you can spend a few weeks to catch up that's not all bad compared to what high-end job offers in other professions demand of your resume.
The more we keep the interview process something demonstrable vs status/prestige/name-check/references, the better.
That's who interviewers are looking for. Now, a case can be made that there aren't that many of those people and you might not need them anyway, but... if you can spend a few weeks to catch up that's not all bad compared to what high-end job offers in other professions demand of your resume.
The more we keep the interview process something demonstrable vs status/prestige/name-check/references, the better.
If you want to receive top compensation, and work at the Meta/Roblox/AirBnBs etc, you will need to be able to solve 2 Leetcode medium questions in 45 minutes, correctly and optimally.
I work for a big tech company where people smell their own farts and think they are geniuses even though they work on CRUD interfaces for admin portals for Cloud. The interview process is well known.
I have seen wrong solutions made by these genius question writers in their own doc, or where they find out 4 weeks later that the time complexity of their own solution is not really what they thhink it is. Do you really think in a company with 50k+ software engineers, where many studied English (but like to conduct CS interviews like they’re CS researchers), that everyone will know and understand details like; how the expected linear time complexity of quickselect works?
It’s a nightmare; it’s blind leading the blind. It’s been completely gamified at this point and the only way to win is to play the game, maybe 10 years ago you could’ve passed on your own merit of having a CS degree and knowing basic DS&A, but I promise you that is almost the case nowhere anymore.
I work for a big tech company where people smell their own farts and think they are geniuses even though they work on CRUD interfaces for admin portals for Cloud. The interview process is well known.
I have seen wrong solutions made by these genius question writers in their own doc, or where they find out 4 weeks later that the time complexity of their own solution is not really what they thhink it is. Do you really think in a company with 50k+ software engineers, where many studied English (but like to conduct CS interviews like they’re CS researchers), that everyone will know and understand details like; how the expected linear time complexity of quickselect works?
It’s a nightmare; it’s blind leading the blind. It’s been completely gamified at this point and the only way to win is to play the game, maybe 10 years ago you could’ve passed on your own merit of having a CS degree and knowing basic DS&A, but I promise you that is almost the case nowhere anymore.
I'm sure they exist, which is great, but I have never had a non-STEM shot on my dev team at any job.
I'm sure you met a few flawed people like you describe. But they are not a large fraction.
I'm sure you met a few flawed people like you describe. But they are not a large fraction.
These jobs pay a lot of money. If you want one bad enough you’ll find the time to practice.
Sure. If what the company doing the hiring is explicitly trying to get employees who are willing to do that.
However if they just want knowledgeable, quality employees, some will self select out of that process.
However if they just want knowledgeable, quality employees, some will self select out of that process.
if that’s the only way you’re able to discern a good candidate from a bad one, that speaks far more loudly about you than any candidate.
> Isn’t that the point? That some people who really don’t want the job won’t invest a bit of their time and people who do really want the job will?
Would you rather have an employee who wants the job, or an employee who is good at the job?
Would you rather have an employee who wants the job, or an employee who is good at the job?
The former if they are good enough to do the job.
jstarfish(5)
[deleted]
making a react component in a 45 minute interview (something I recently did) is better than leetcode, but still a poor signal. on the job I'll never do a time trial where I'm being watched and gaslit about "just wanting to see how I think" as opposed to "do it perfectly from memory in this collaborative coding session faster than this guy escaping a dystopia in Asia, and trying not to be deported back"
find a way to simulate that I will do it some time over the next two weeks because the product manager reduced everyone's story points in half for this sprint in the grooming session, and the fact that I even did that gets me praised because nobody else on the team finished their tasks
find a way to simulate that I will do it some time over the next two weeks because the product manager reduced everyone's story points in half for this sprint in the grooming session, and the fact that I even did that gets me praised because nobody else on the team finished their tasks
> faster than this guy escaping a dystopia in Asia, and trying not to be deported back
While that guy may have an advantage over you in motivation, you might have an advantage over him in communication.
While that guy may have an advantage over you in motivation, you might have an advantage over him in communication.
Also known as accent.
And grammar, vocabulary, tone, etiquette… a lot more goes into communication than accent alone.
That's called an in-basket test.[1]
For a really unusual interview process, see how Jane Street, the trading firm, does it. They expect people to do well at this.[2]
Their actual interview process starts each candidate with a stack of poker chips. In each interview, candidates get to make bets on various things that look like financial trades. If they run out of chips, they're out.
[1] https://en.wikipedia.org/wiki/In-basket_test
[2] https://figgie.com/how-to-play
For a really unusual interview process, see how Jane Street, the trading firm, does it. They expect people to do well at this.[2]
Their actual interview process starts each candidate with a stack of poker chips. In each interview, candidates get to make bets on various things that look like financial trades. If they run out of chips, they're out.
[1] https://en.wikipedia.org/wiki/In-basket_test
[2] https://figgie.com/how-to-play
A software system differs from the chaos the trading world represents. You can try to evaluate traders, but the deep truth is that very good traders can become very bad traders seamlessly because nothing guarantees any success of any kind in that domain.
When interviewing software engineers you shouldn't care that much about the abilities that someone shows during the time of a technical interview on a specific day; that's not an appropriate way to measure the skills you need for your business. Leet code is an appropriate way to measure how much such engineer is able to perform as a competitive coder. You will never need a competitive coder in your team; because what to be done in SE is not about coding less than 10 lines of code in less than X minutes to transform data A into data B. But back to candidate selection, due to the short amount of time you have to meet applicants, you finally decide to hire whatever person that succeeded to find the fastest solution to the almost scholastic problem you proposed because in the end, you fail to understand what could be the appropriate selection method : aren't all candidates more or less equivalent if you stop proposing leet code problems ? What can prove that X person with 5 years of exp is less good than that one that worked at google for 6 years ? Does it really matters than candidate Y doesn't know Go when all the languages we use aren't that truly different ? Is someone from nowhere, without any education of any sort, but that seems to be an hobbyist programmer since his childhood really a bad candidate ?
You never know, but because whether you're HR or the person with technical authority in the team, you don't want to risk anything, and you believe that a good leet code solver can do the work.
And finally, to me, that's the truth : more than 50% of the persons you decided to offer an interview would have done the work with success, and at least 20% seemed very nice, and you couldn't choose so leet code was the thing.
They should live stream interviewing the stock trading fish. Go pro, little dude!
On average, I enjoy these kinds of interviews much more than leetcode. The quality of the prep work and the ad libbing makes a huge difference to the quality of the interview though. One particularly memorable interview involved verbally debugging intermittent crashes the interviewer had recently seen. Unfortunately, they had no information about the software stack, unique symptoms, or access to status registers so it ended up like:
"Is this bit of information available?"
"No"
Repeated for 30 minutes.
"Is this bit of information available?"
"No"
Repeated for 30 minutes.
That seems like an interview that you passed (the company learned that you know a lot more than them about the resources available for debugging).
It also sounds like an interview that the company failed.
More likely the interviewer was some internal recruiter type who had been given some example questions to ask but didn't know about development themselves.
They were an engineer on the team I would have led, not a recruiter.
Also a high level of frustration tolerance.
In interviewing I try to avoid approaches that in any form or shape resemble riddles, so ostensibly open problems but with predetermined acceptable answers.
I've found that it's both frustrating for the candidate and doesn't give me much insight.
Such a back and forth to me is too close for comfort, even though I used to be into role-playing games back in the day.
Currently I try to use the little time I have to determine whether the person in front of me is a charlatan, so only questions that are easy to answer for someone actually having the skills advertised in their CV.
I've found that it's both frustrating for the candidate and doesn't give me much insight.
Such a back and forth to me is too close for comfort, even though I used to be into role-playing games back in the day.
Currently I try to use the little time I have to determine whether the person in front of me is a charlatan, so only questions that are easy to answer for someone actually having the skills advertised in their CV.
Could you just have them sign an NDA and pair with people on some non-trade-secret real issue that needs addressing?
If you start by giving a little bit of context and then jump into some reasonable end of the backlog, then you get to see how they think of priorities, they get to ask you lots of contextual questions, you get negotiate with each other a thing to do, you get to investigate and/or develop a solution, and at the end of it all parties involved got lots of signal about what it’s like and what’s required to do the job. Not to mention you stand the chance of some normally useful thing getting further along too. Work with the person like they’re “consultant for a day” and see where you get to.
Why is our industry so resolute to come up with a bunch of bespoke, oblique representations of a job when the actual job is right there available to be used for assessment?
If you start by giving a little bit of context and then jump into some reasonable end of the backlog, then you get to see how they think of priorities, they get to ask you lots of contextual questions, you get negotiate with each other a thing to do, you get to investigate and/or develop a solution, and at the end of it all parties involved got lots of signal about what it’s like and what’s required to do the job. Not to mention you stand the chance of some normally useful thing getting further along too. Work with the person like they’re “consultant for a day” and see where you get to.
Why is our industry so resolute to come up with a bunch of bespoke, oblique representations of a job when the actual job is right there available to be used for assessment?
My favorite interview experience was a short take-home that required me to write a toy version of something the company actually did on a day-to-day basis, followed by an in-person pairing session to extend that toy to handle more functionality and cover certain edge cases.
Of course, the downside is that it required me to spend four hours working on this at home, and then another several hours at the interview - but having to spend time interviewing is something that cannot be avoided, so I'd much rather the time be spent on something relevant that actually shows off my abilities, rather than playing Advent of Code with money at stake, or filling out astrological profiles.
Of course, the downside is that it required me to spend four hours working on this at home, and then another several hours at the interview - but having to spend time interviewing is something that cannot be avoided, so I'd much rather the time be spent on something relevant that actually shows off my abilities, rather than playing Advent of Code with money at stake, or filling out astrological profiles.
What hourly rate did you charge?
I have bills, and free labor doesn't pay them. If an employer spends 1-2 hours talking tech, the role, experience, culture fit, and maybe a quick "yes, I can code" toy PoC -- fine, I'll volunteer. But if one interview process requires a day+ of investment...I'll pass, thanks. Ditto for whiteboard hazing rituals, whose prerequisites include memorizing algo' arcana and cute puzzles. I have personal obligations, which I won't sacrifice to practice dancing like a clown for a stranger with a fetish for reinventing bubble sort or whatever.
Either respect my time and status as your peer, or don't bother wasting both of our time. If that means fewer opportunities for me, then so be it. My deathbed reflections won't include "Damn, I regret spending so much time with loved ones instead of learning how to optimally stack rings onto three pegs."
Apologies for the rant. I'm clearly passionate about this lol
I have bills, and free labor doesn't pay them. If an employer spends 1-2 hours talking tech, the role, experience, culture fit, and maybe a quick "yes, I can code" toy PoC -- fine, I'll volunteer. But if one interview process requires a day+ of investment...I'll pass, thanks. Ditto for whiteboard hazing rituals, whose prerequisites include memorizing algo' arcana and cute puzzles. I have personal obligations, which I won't sacrifice to practice dancing like a clown for a stranger with a fetish for reinventing bubble sort or whatever.
Either respect my time and status as your peer, or don't bother wasting both of our time. If that means fewer opportunities for me, then so be it. My deathbed reflections won't include "Damn, I regret spending so much time with loved ones instead of learning how to optimally stack rings onto three pegs."
Apologies for the rant. I'm clearly passionate about this lol
Couldn't agree more; if the exercise takes more than 20-30 minutes, I am always asking if they are ok with me charging them my freelance hourly rate for the time needed to complete the exercise (regardless of hiring result).
I mean, sure, that's fine. But if you discount the take-home portion - four hours - everything else would still have taken time, because I had to spend time on the phone with HR for the initial screen, and I had to go in for an interview afterwards anyway, and ain't nobody going to pay you for the privilege of interviewing. And if I'm going to spend time displaying my abilities, doing it quietly at home on my own time is vastly better than in a shitty meeting room with two people breathing down my fucking neck.
Do take-homes suck, and unfairly privilege people with more free time? Yes. Are they slightly better than the other asinine bullshit interviewers love to throw at candidates? Fucking absolutely.
Do take-homes suck, and unfairly privilege people with more free time? Yes. Are they slightly better than the other asinine bullshit interviewers love to throw at candidates? Fucking absolutely.
Unfortunately I'm a corporate drone, so I don't have even nearly enough control over the process to make anyone sign anything.
I think one issue with this is that some candidates who might be great at the actual job could, for whatever reason, be bad at this verbal type of simulation. Perhaps they work by staring at the problem on their screen a lot, perhaps they struggle to think at the same time as roleplaying. Just saying that this roleplaying aspect is somewhat tested for and doesnt align with the job exactly.
The interview the post describes actually sounds like a good format for me, but I do have some problems adjacent to the ones you mention.
As someone who struggles with ~parallelizing some tasks (especially when anxious), it would be great to have an up-front check in about how comfortable I am with the format of any technical ~challenge. (And even better to discover there's some wiggle room to accommodate.)
For example, I find it very difficult to code my way through a problem and narrate what I'm doing and take feedback while I know I'm being evaluated and the clock's ticking. If the interviewer says something important in the middle of this, I often have to put down any plates I've managed to get spinning (and maybe go offload some adrenaline) and then ask them to repeat before I can even parse the words they said, let alone figure out how they apply to the problem.
As someone who struggles with ~parallelizing some tasks (especially when anxious), it would be great to have an up-front check in about how comfortable I am with the format of any technical ~challenge. (And even better to discover there's some wiggle room to accommodate.)
For example, I find it very difficult to code my way through a problem and narrate what I'm doing and take feedback while I know I'm being evaluated and the clock's ticking. If the interviewer says something important in the middle of this, I often have to put down any plates I've managed to get spinning (and maybe go offload some adrenaline) and then ask them to repeat before I can even parse the words they said, let alone figure out how they apply to the problem.
A lot of companies would much rather pass over a great fit than hire a bad fit. Not hiring great people isn’t that bad of a thing in most cases.
Yes precisely. Make sure that you’re actually selecting for what you mean to be selecting for.
This sounds like a good exercise, and kudos for thinking differently and at least trying to create a fairer and more useful process.
Two comments I'd make, that I hope you are already thinking about:
1. Role playing is a cringe-inducing experience for many (most?) adults. Please credit the interviewee with intelligence and don't try to actually role-play this. These are hypothetical scenarios, so discuss them as such.
2. As interviewers we are incredibly prone to seeing the things we know as obvious. When you devise an exercise, you can start to believe that everyone should easily see what you consider to be the 'right' way forward. In fact, seeing it may rely on your many years of experience in your company's culture, or your many years of experience with the interview exercise! So please, always remain skeptical of your assumptions about what is obvious, and avoid the trap of expecting candidates to see or follow the path that you think it's best. If any of your support scenarios are actually devious riddles that require one perfect question or leap, then whether candidates 'get it' or not will be random rather than indicative of better performance in-post.
Two comments I'd make, that I hope you are already thinking about:
1. Role playing is a cringe-inducing experience for many (most?) adults. Please credit the interviewee with intelligence and don't try to actually role-play this. These are hypothetical scenarios, so discuss them as such.
2. As interviewers we are incredibly prone to seeing the things we know as obvious. When you devise an exercise, you can start to believe that everyone should easily see what you consider to be the 'right' way forward. In fact, seeing it may rely on your many years of experience in your company's culture, or your many years of experience with the interview exercise! So please, always remain skeptical of your assumptions about what is obvious, and avoid the trap of expecting candidates to see or follow the path that you think it's best. If any of your support scenarios are actually devious riddles that require one perfect question or leap, then whether candidates 'get it' or not will be random rather than indicative of better performance in-post.
> Role playing is a cringe-inducing experience for many (most?) adults.
Depends a bit on the how: I don't think most find fire exercises, military exercises cringy - although they are a form of roleplay?
Depends a bit on the how: I don't think most find fire exercises, military exercises cringy - although they are a form of roleplay?
A fire drill where we don't pretend it's real but do take the drill seriously, is fine.
A fire drill where the Health & Safety guy insists that we pretend it's real - yes, most people find this cringe-inducing. For more on this, see Dwight Schrute.
A fire drill where the Health & Safety guy insists that we pretend it's real - yes, most people find this cringe-inducing. For more on this, see Dwight Schrute.
There's a bit of a difference between a fire drill in an office building from the perspective of a regular office employee - and a firefighter. In the latter case it would be expected to have people working as victims complete with mocha[1] injuries and feigned panic.
I don't see why simulating working a bug report need be cringe inducing... If everyone is just playing themselves.
[1] unintentional humor. I suppose there are some unfortunate people that have had to simulate helping someone with a mocha burn or similar injury. I meant "mock injury".
I don't see why simulating working a bug report need be cringe inducing... If everyone is just playing themselves.
[1] unintentional humor. I suppose there are some unfortunate people that have had to simulate helping someone with a mocha burn or similar injury. I meant "mock injury".
> There's a bit of a difference between a fire drill in an office building from the perspective of a regular office employee - and a firefighter
And which one of those do you think a tech support interview is closer to?
And which one of those do you think a tech support interview is closer to?
It is possible to structure interviews so that:
* Every candidate get an almost-identical interview
* The interview is not heavily dependent on the investment or natural aptitude of the interviewer
* Interviews generate data that can be scored by a rubric
* Interviews aren't scored by the interviewer, thus eliminating the intrinsic bias of being in the hot seat with the candidate
It looks different for every role. For instance, for a penetration testing role, I've run standardized interviews that involve generating and prioritizing attack surface lists and hypothetical bug lists. The output isn't "pass/fail", it's the list of places to test, the list of bugs expected, scheduling/testing budgets, and things like that. If I'm delivering the interview to the candidate, I'm presenting the panel of reviewers not my opinions on the candidate, but what they came up with.
The same approach works for general software development (we don't use it here, we have an even fussier process). You can do design exercises, you can work out the dependencies that are going to be needed for a particular complicated project, you can do estimation exercises, you can catalog likely bugs. Review some PRs together. Look at the day-to-day work your team does, and then make the exercise a model of that work.
It's not especially easy to do compared to just sitting down and asking a bunch of questions. Moreover, it's very hard to adopt it as a practice, because you have to trust the rubric (iterating on it over time, but using it enough for it to be meaningful) and not override it based on extrinsic considerations (referrals, misgivings about candidate background, &c).
But the upside is pretty obvious? Once you have it worked out, you get a repeatable process, something that is naturally geared towards iteration and improvement.
Standardize your interviews, generate data, have a panel make the decisions and not the interviewer.
You can still play the interview like D&D if you want!
* Every candidate get an almost-identical interview
* The interview is not heavily dependent on the investment or natural aptitude of the interviewer
* Interviews generate data that can be scored by a rubric
* Interviews aren't scored by the interviewer, thus eliminating the intrinsic bias of being in the hot seat with the candidate
It looks different for every role. For instance, for a penetration testing role, I've run standardized interviews that involve generating and prioritizing attack surface lists and hypothetical bug lists. The output isn't "pass/fail", it's the list of places to test, the list of bugs expected, scheduling/testing budgets, and things like that. If I'm delivering the interview to the candidate, I'm presenting the panel of reviewers not my opinions on the candidate, but what they came up with.
The same approach works for general software development (we don't use it here, we have an even fussier process). You can do design exercises, you can work out the dependencies that are going to be needed for a particular complicated project, you can do estimation exercises, you can catalog likely bugs. Review some PRs together. Look at the day-to-day work your team does, and then make the exercise a model of that work.
It's not especially easy to do compared to just sitting down and asking a bunch of questions. Moreover, it's very hard to adopt it as a practice, because you have to trust the rubric (iterating on it over time, but using it enough for it to be meaningful) and not override it based on extrinsic considerations (referrals, misgivings about candidate background, &c).
But the upside is pretty obvious? Once you have it worked out, you get a repeatable process, something that is naturally geared towards iteration and improvement.
Standardize your interviews, generate data, have a panel make the decisions and not the interviewer.
You can still play the interview like D&D if you want!
[deleted]
I went through something similar once when I was interviewing for an IT position in the Antarctic (I'd applied for positions at both McMurdo and Amundsen-Scott), and I was supposed to connect to their network simulator to troubleshoot and solve some network issue, but it wasn't working on their end so the interviewer walked me through the scenario, and responded as a DM would. - Quite frustratingly this was my 3rd interview for the position, and although I diagnosed and solved the DM'd scenario of a downed vlan issue in a Cisco environment in ~5 minutes of this off-the-cuff "simulation" using DM provided prompts, help menus, and outputs, I did not get the job. I think he penalized me for relying on the DM provided help menus and auto suggest tabbing...which is something anyone on a working simulator would have done. Which was a bummer because even though I was overqualified for the positions and would take a pay cut, I really wanted the experience of living for months in the Antarctic. So while I see the usefulness of this kind of interview, there are noteworthy issues of bias which a real simulator wouldn't begrudge you.
I think the "fake product" interview is a great format, but it can be done poorly if the setup is unrealistic. I've been on the receiving end of a couple of very bad attempts.
In one, which I have described here before, an interviewer asked me to design a system for selling concert tickets. After half an hour of confused back and forth, it emerged that they wanted a network service that replied to requests with unpredictable random integers, which did not repeat, unless the service rebooted and then it was okay if there was an accidental repeat. Needless to say, I started with a lot of assumptions based on the idea of selling concert tickets, and I don't think it reflected poorly on me that it took so long to figure out that the real problem they were judging me by was so radically different from the one they asked me to solve.
I had another interview recently which had a similar wrinkle. The interviewer asked for a system to solve a very particular problem, which because of the particulars of the problem would involve receiving very small amounts of data from systems in the field. The interviewer asked multiple times about the cost of data ingress and data storage, and each time I deflected them, saying that it wasn't significant compared to other costs in the system. I was so focused on solving the problem as stated that it didn't dawn on me until later that the interviewer likely had a different problem in mind in which the data was much bigger, and they were evaluating whether I could be sensitive to those costs and design an economical system.
Interviews like these are very frustrating, and I don't think they will lead to great hires. You will end up hiring people who solve the problem you have in mind even though you told them to solve a different problem. These programmers will have the same biases and preoccupations as you, which might make them easy to incorporate into your team, but they will probably have the same blind spots as well, which means your team will be less prepared to solve new problems in the future.
In one, which I have described here before, an interviewer asked me to design a system for selling concert tickets. After half an hour of confused back and forth, it emerged that they wanted a network service that replied to requests with unpredictable random integers, which did not repeat, unless the service rebooted and then it was okay if there was an accidental repeat. Needless to say, I started with a lot of assumptions based on the idea of selling concert tickets, and I don't think it reflected poorly on me that it took so long to figure out that the real problem they were judging me by was so radically different from the one they asked me to solve.
I had another interview recently which had a similar wrinkle. The interviewer asked for a system to solve a very particular problem, which because of the particulars of the problem would involve receiving very small amounts of data from systems in the field. The interviewer asked multiple times about the cost of data ingress and data storage, and each time I deflected them, saying that it wasn't significant compared to other costs in the system. I was so focused on solving the problem as stated that it didn't dawn on me until later that the interviewer likely had a different problem in mind in which the data was much bigger, and they were evaluating whether I could be sensitive to those costs and design an economical system.
Interviews like these are very frustrating, and I don't think they will lead to great hires. You will end up hiring people who solve the problem you have in mind even though you told them to solve a different problem. These programmers will have the same biases and preoccupations as you, which might make them easy to incorporate into your team, but they will probably have the same blind spots as well, which means your team will be less prepared to solve new problems in the future.
To be fair, I would say if your task is to design a system it's your job to find out what the person actually wants and not run with assumptions you made based on a bad description. The reality is that you will usually start with a bad description. So I'm not sure if these misunderstandings actually reflect that badly on the interviewer/the question.
I think if they were testing my ability to work through misunderstandings in requirements, they would have been decently impressed that I got from "system for selling concert tickets" to "system for giving out random integers" in half an hour. But they weren't impressed, they were irritated.
Edit: Also I don't think a normal product requirements discovery process involves figuring out that you're actually being asked to solve a different problem in a vastly different domain. You shouldn't ask someone to build Netflix while thinking "let's see how long it takes them to realize I meant Twitter."
Edit: Also I don't think a normal product requirements discovery process involves figuring out that you're actually being asked to solve a different problem in a vastly different domain. You shouldn't ask someone to build Netflix while thinking "let's see how long it takes them to realize I meant Twitter."
Here's how to recruit software developers.
1: define the requirements for the job
2: devise questions that attempt to give insight into how well the candidate might meet that requirement
3: note down the candidates suitability out of ten for each question
4: TALK to the candidate in a free flowing and open discussion about their work, what they have done, how they built it, why they built it that way, their role in the project, what went right, what went wrong, what they would do differently.
5: Whether or not the candidate asks questions is a meaningless measure. What is meaningful is to sense their level of curiosity - about this job that you are offering, about computing and technology in general. Software development is hard and being good at it requires a level of curiosity. It's probably a concern if someone shows no curiosity in this job or software, though you need to be cautious about how you measure this - did you do a good enough job to explore their level of curiosity? Curiosity does not necessarily reveal itself without careful probing.
6: Discuss the actual work of this job with the candidate. Quickly run through the actual todo list of things that need to be done right now - talk through the hardest of those tasks - how would they approach it?
7: take a leap and employ the person you feel is right - do NOT go into ever deeper analysis, ever more interviews, ever more tests and checks and hoops. No matter how much interviewing you do, you probably won't really know how someone goes in this job until they've been in it 3 or more months.
8: Don't be so risk averse that you don't hire anyone. Do it based on your best guess. it might sound harsh, but do your best and be willing to fire them before their 3 month trial is up if they are not the right person.
9: Take into account enthusiasm - how much does this person want this job? And no, do NOT measure this until the end of the interview process - looking for enthusiasm in a cover letter is like wanting a girlfriend/boyfriend commitment before first date.
10: Pay more than people expect, if you can. $1K less than someone says they want/need reduces first day enthusiasm for the job, $1K more increases first day enthusiasm. Wind the numbers up and down for greater impact.
11: After the interview process is done, talk to referees, analyses interview results apples versus apples.
12: Interview gimmicks, leetcode, abstract questions should be red flags for interviewees. If you're looking for a job and find yourself in a silly interview, be willing to politely call it off and save yourself the time and hassle of dealing with a company that does not know how to recruit software developers.
1: define the requirements for the job
2: devise questions that attempt to give insight into how well the candidate might meet that requirement
3: note down the candidates suitability out of ten for each question
4: TALK to the candidate in a free flowing and open discussion about their work, what they have done, how they built it, why they built it that way, their role in the project, what went right, what went wrong, what they would do differently.
5: Whether or not the candidate asks questions is a meaningless measure. What is meaningful is to sense their level of curiosity - about this job that you are offering, about computing and technology in general. Software development is hard and being good at it requires a level of curiosity. It's probably a concern if someone shows no curiosity in this job or software, though you need to be cautious about how you measure this - did you do a good enough job to explore their level of curiosity? Curiosity does not necessarily reveal itself without careful probing.
6: Discuss the actual work of this job with the candidate. Quickly run through the actual todo list of things that need to be done right now - talk through the hardest of those tasks - how would they approach it?
7: take a leap and employ the person you feel is right - do NOT go into ever deeper analysis, ever more interviews, ever more tests and checks and hoops. No matter how much interviewing you do, you probably won't really know how someone goes in this job until they've been in it 3 or more months.
8: Don't be so risk averse that you don't hire anyone. Do it based on your best guess. it might sound harsh, but do your best and be willing to fire them before their 3 month trial is up if they are not the right person.
9: Take into account enthusiasm - how much does this person want this job? And no, do NOT measure this until the end of the interview process - looking for enthusiasm in a cover letter is like wanting a girlfriend/boyfriend commitment before first date.
10: Pay more than people expect, if you can. $1K less than someone says they want/need reduces first day enthusiasm for the job, $1K more increases first day enthusiasm. Wind the numbers up and down for greater impact.
11: After the interview process is done, talk to referees, analyses interview results apples versus apples.
12: Interview gimmicks, leetcode, abstract questions should be red flags for interviewees. If you're looking for a job and find yourself in a silly interview, be willing to politely call it off and save yourself the time and hassle of dealing with a company that does not know how to recruit software developers.
> 12: Interview gimmicks, leetcode, abstract questions should be red flags for interviewees. If you're looking for a job and find yourself in a silly interview, be willing to politely call it off and save yourself the time and hassle of dealing with a company that does not know how to recruit software developers.
The world doesn't work like that. Most companies are going to put you through the 10 round process or some other tedious process. Unless you're financially willing and able to turn down over 50% (probably a lot more actually) of job opportunities. And most people have bills, mortgages, rent, food they'd like to be able to afford. So it goes.
I don't like it either btw. I agree with you in principle.
Rest of your points are good though.
The world doesn't work like that. Most companies are going to put you through the 10 round process or some other tedious process. Unless you're financially willing and able to turn down over 50% (probably a lot more actually) of job opportunities. And most people have bills, mortgages, rent, food they'd like to be able to afford. So it goes.
I don't like it either btw. I agree with you in principle.
Rest of your points are good though.
I wish you were the recruiter of every company in the world.
[deleted]
Love this approach! We used this method for awhile when conducting interviews, usually with a couple problems escalating in difficulty. Found it to be way more helpful to watch how someone troubleshoots than to test their aptitude with specific skills
This is called simulation and it is a useful tool both when selecting for and training for cognitive skills.
Importantly and usefully, simulations for cognitive work do not require any sort of physical fidelity, so it's really easy to set up basic scenarios like for hiring.
(The main cost in using simulation for training lies in designing scenarios that train for expertise. That takes slightly more advanced elicitation methods which fall under the umbrella of cognitive task analysis.)
Importantly and usefully, simulations for cognitive work do not require any sort of physical fidelity, so it's really easy to set up basic scenarios like for hiring.
(The main cost in using simulation for training lies in designing scenarios that train for expertise. That takes slightly more advanced elicitation methods which fall under the umbrella of cognitive task analysis.)
I've just went through a round of interviews from various companies after being made redundant from work 4 months ago.
Admittedly the job market was really quiet but it was eye opening the differences in interview techniques and processes.
I had one interview from a FinTech company which I actually thought went quite well, the next day I got a rejection stating "I need to improve my STAR technique". I just rolled my eyes thinking FFS
Another company the CTO wasn't on the call but had asked to record the last 15 minutes of the interview with questions he'd prepared, I was close to terminating the interview. If he couldn't be assed sitting in on the interview why should I answer questions the interviewer himself didn't understand. One of the questions was "What is triple D". I'm an experienced dev, fully aware there are too many acronyms in the industry but had never heard of triple D before, when I googled it after I was thinking FFS, that's just what any competent dev does by default. I guesses data driven development but admitted I wasn't sure.
Another interview the principal engineer yawned 3 times when I was talking before I even got to the half way stage, not one apology. I know it's just human nature but to not even acknowledge he was making me uncomfortable reminded me afterwards that it's probably a toxic work culture, which I've been told since is the case from people who worked there.
Thankfully at the start of the new year I was offered 3 roles, 2 of them I thought I'd screwed up the interview. The one I accepted, apart from taking on a task to review some code and raise issues with it, I was asked to describe an architecture of some system I'd worked on and enjoyed. I spoke too long, going past the interview time but didn't feel I explained the whole system.
I understand interviewing candidates is difficult, I've had to do it a few times in the past but the competency of interviewers and the process to score candidates varies wildly from organisation to organisation. A realisation for me although I was already aware of it, is the personality of the people interviewing you varies wildly, more often than not it's a good indication of the organisation itself.
I had one interview from a FinTech company which I actually thought went quite well, the next day I got a rejection stating "I need to improve my STAR technique". I just rolled my eyes thinking FFS
Another company the CTO wasn't on the call but had asked to record the last 15 minutes of the interview with questions he'd prepared, I was close to terminating the interview. If he couldn't be assed sitting in on the interview why should I answer questions the interviewer himself didn't understand. One of the questions was "What is triple D". I'm an experienced dev, fully aware there are too many acronyms in the industry but had never heard of triple D before, when I googled it after I was thinking FFS, that's just what any competent dev does by default. I guesses data driven development but admitted I wasn't sure.
Another interview the principal engineer yawned 3 times when I was talking before I even got to the half way stage, not one apology. I know it's just human nature but to not even acknowledge he was making me uncomfortable reminded me afterwards that it's probably a toxic work culture, which I've been told since is the case from people who worked there.
Thankfully at the start of the new year I was offered 3 roles, 2 of them I thought I'd screwed up the interview. The one I accepted, apart from taking on a task to review some code and raise issues with it, I was asked to describe an architecture of some system I'd worked on and enjoyed. I spoke too long, going past the interview time but didn't feel I explained the whole system.
I understand interviewing candidates is difficult, I've had to do it a few times in the past but the competency of interviewers and the process to score candidates varies wildly from organisation to organisation. A realisation for me although I was already aware of it, is the personality of the people interviewing you varies wildly, more often than not it's a good indication of the organisation itself.
I was expecting different dungeon master: https://thefinalrumble.miraheze.org/wiki/Van_Darkholme
Took me a moment or two to realize this is about DnD.
After all, interviews tend to be a stressful and intense process...
After all, interviews tend to be a stressful and intense process...
A good interview should be a good fit for both parties. Just have a casual lunch with them. You should be able to tease out if they are a good fit. These “hacks” are so ridiculous. Way to overthink something.
> it feels like a very, very light version of the prep you do when designing a campaign as a DM
With todays chatgpts, that reads like a pretty decent prep for a campaign as a DM.
With todays chatgpts, that reads like a pretty decent prep for a campaign as a DM.
wow awesome analogy. I got rejected because my leet code solution wasn't fast enough for a security engineer position. The person who referred me was furious. Meanwhile, another interviewer did just this to me and it let me flex my knowledge.
Would love to see this more. There is also the inevitable future where someone publishes a book on it and it gets cargo-culted. C'est la vie.
Would love to see this more. There is also the inevitable future where someone publishes a book on it and it gets cargo-culted. C'est la vie.
That actually seems like a good idea. Interesting, practical and you can see if they are socially inept. I’d actually try this.
That's very similar to how good product management interviews are conducted.
Companies should rather ask the candidate to describe a project they completed.
Let the interviewer then dive into any technical challenges such a project may propose, in order to vet technical skills.
This allows a chance for candidate to show they can connect high level big picture thinking with low level technical skills.
Let the interviewer then dive into any technical challenges such a project may propose, in order to vet technical skills.
This allows a chance for candidate to show they can connect high level big picture thinking with low level technical skills.
This sounds like a regular situational interview?
I'm curious what changed around 2015-2016 that led to the current interview process. In the before times the interview was more of a technical conversation, sure they had some gotcha questions, but nothing too brutal. If you had real experience that you could talk about, maybe it was validated against references or a background check. People usually looked for a CS or adjacent degree, and that was enough. The quality of folks wasn't too different in my experience.
I really feel for those with a lot of experience that are being dropped into the fire, especially those with families who may not have the time to memorize patterns for 3 months, but are still very capable. Non-tech, low paying jobs now ask you to create a soduko solver or trap rain water within 45 minutes. There's not many places to go for those that don't 'adapt'.