They don't even know the fundamentals(blog.royalsloth.eu)
blog.royalsloth.eu
They don't even know the fundamentals
https://blog.royalsloth.eu/posts/they-dont-even-know-the-fundamentals/
322 comments
The author says, "I wouldn’t be able to explain from the top of my head what the ACID term means".
I think there is a big difference between not knowing something off the top of your head and being completely unaware of it.
If you have awareness of something, you can make decisions based on that awareness without necessarily having a deep understanding. When you have no awareness is when the big issues arise.
For example, with your MongoDB fuckup if you had even a vague idea about the underlying behaviour you might have been able to consider the failure scenario ahead of time and prevent it from occurring. A deep understanding of how MongoDB works probably wasn't required to prevent this incident.
Same for indexes. If you're aware of them, you can trigger the thought process of "this query is pretty slow, maybe I should look into adding an index because I know that speeds things up". You don't necessarily have to understand how a B-tree works for an index to be effective.
I think there is a big difference between not knowing something off the top of your head and being completely unaware of it.
If you have awareness of something, you can make decisions based on that awareness without necessarily having a deep understanding. When you have no awareness is when the big issues arise.
For example, with your MongoDB fuckup if you had even a vague idea about the underlying behaviour you might have been able to consider the failure scenario ahead of time and prevent it from occurring. A deep understanding of how MongoDB works probably wasn't required to prevent this incident.
Same for indexes. If you're aware of them, you can trigger the thought process of "this query is pretty slow, maybe I should look into adding an index because I know that speeds things up". You don't necessarily have to understand how a B-tree works for an index to be effective.
> If you have awareness of something, you can make decisions based on that awareness without necessarily having a deep understanding. When you have no awareness is when the big issues arise.
I think in many cases, it's that even just having a vague awareness of something prompts you to double-check more often or to do something about it when you aren't really sure.
As an example, I find myself reading about the pros and cons of various PostgreSQL index types over and over again. It isn't really all that important for me to retain every detail about them for immediate recall but knowing that there's enough of a difference motivates me to read the documentation or run a quick web search before shooting myself in the foot.
I tend to think that testing people on this kind of deep knowledge isn't really helpful nor is it indicative of how developers really operate. What you need to test them on is what steps they would take when they aren't really sure and how they learn from it.
I think in many cases, it's that even just having a vague awareness of something prompts you to double-check more often or to do something about it when you aren't really sure.
As an example, I find myself reading about the pros and cons of various PostgreSQL index types over and over again. It isn't really all that important for me to retain every detail about them for immediate recall but knowing that there's enough of a difference motivates me to read the documentation or run a quick web search before shooting myself in the foot.
I tend to think that testing people on this kind of deep knowledge isn't really helpful nor is it indicative of how developers really operate. What you need to test them on is what steps they would take when they aren't really sure and how they learn from it.
> The author says, "I wouldn’t be able to explain from the top of my head what the ACID term means".
> I think there is a big difference between not knowing something off the top of your head and being completely unaware of it.
The next level is being aware that the “I” doesn't quite apply in most databases with their default transaction settings, true isolation/serialisability being sacrificed for performance, though the distinction rarely matters is can be significant.
> You don't necessarily have to understand how a B-tree works for an index to be effective.
How a b-tree operates can be safely kept as a black-box thing, the important level of detail that many miss in my experience is how compound indexes can (and more importantly, can't) be used. Others include that in most databases defining a foreign key relationship does not create a supporting index by default, and that careless use of functions in joining & filtering clauses often blocks index use. If you have a DB specialist on the team there is probably no need for other devs to be intimately aware of even these details, but they should at least be aware enough of their existence than they know to ask the local expert (or look things up elsewhere) to check their working when the issues might affect what they are working on.
As a bit of a local DB specialist who has met some bad ones, I should point out that it is important that we, and other local “exports”, be approachable so the other devs feel they can run things by us for verification/improvement.
The next level is being aware that the “I” doesn't quite apply in most databases with their default transaction settings, true isolation/serialisability being sacrificed for performance, though the distinction rarely matters is can be significant.
> You don't necessarily have to understand how a B-tree works for an index to be effective.
How a b-tree operates can be safely kept as a black-box thing, the important level of detail that many miss in my experience is how compound indexes can (and more importantly, can't) be used. Others include that in most databases defining a foreign key relationship does not create a supporting index by default, and that careless use of functions in joining & filtering clauses often blocks index use. If you have a DB specialist on the team there is probably no need for other devs to be intimately aware of even these details, but they should at least be aware enough of their existence than they know to ask the local expert (or look things up elsewhere) to check their working when the issues might affect what they are working on.
As a bit of a local DB specialist who has met some bad ones, I should point out that it is important that we, and other local “exports”, be approachable so the other devs feel they can run things by us for verification/improvement.
Yeah that’s the difference between a “blank stare” in the interview and a decent response. Anyone applying for a database position should know that relates to transactions and yadda-yadda. They should know why transactions are important, etc.
But… I’d like to say this is the kind of problem these quick-boot coding schools fall short on. Sure one can learn programming language X, but that doesn’t mean one has any sort of fundamentals in algorithms, data structures or general database technology.
But… I’d like to say this is the kind of problem these quick-boot coding schools fall short on. Sure one can learn programming language X, but that doesn’t mean one has any sort of fundamentals in algorithms, data structures or general database technology.
> this query is pretty slow, maybe I should look into adding an index because I know that speeds things up
On my current work database is riddled with indices over various status columns. Those indices are not selective and basically useless. Worse of all: it's not immediately obvious that those indices are bad, because inserts/updates are fast enough, but they slow system down as they accumulate and there's no easy way back.
On my current work database is riddled with indices over various status columns. Those indices are not selective and basically useless. Worse of all: it's not immediately obvious that those indices are bad, because inserts/updates are fast enough, but they slow system down as they accumulate and there's no easy way back.
Sadly, a good DBA would be able to give you information on these things, but the DBA like system admins is a fading "not needed" position. I know HN hates stored procedures, but having an application use stored procedures instead of on-the-fly queries allows a team to do the proper optimizations. Databases evolve with use and added records, and few teams devote resources to stay ahead of the changes.
I personally love stored procedures, and Postgres, MSSQL, Sybase and Oracle all provide pretty good languages to implement them in.
In my experience the most reliable and long living projects treat the DB as its own separate product, whose interface to data mutation is via stored procs. The db can then be sanely designed with good data integrity rules (foreign keys, unique indexes and data column constraints).
Then external callers use the stored procs as their API, and then mostly don't have to worry about the internals. You can easily create a web based client, REST calls or a native GUI app that all work directly off the same db.
However they do have to worry about : -connection timeouts -query timeouts -deadlocks
But that's the same for all distributed systems.
In my experience the most reliable and long living projects treat the DB as its own separate product, whose interface to data mutation is via stored procs. The db can then be sanely designed with good data integrity rules (foreign keys, unique indexes and data column constraints).
Then external callers use the stored procs as their API, and then mostly don't have to worry about the internals. You can easily create a web based client, REST calls or a native GUI app that all work directly off the same db.
However they do have to worry about : -connection timeouts -query timeouts -deadlocks
But that's the same for all distributed systems.
> For example, with your MongoDB fuckup if you had even a vague idea about the underlying behaviour you might have been able to consider the failure scenario ahead of time and prevent it from occurring.
This is going to require being conversant with the terms in that ACID acronym.
> A deep understanding of how MongoDB works probably wasn't required to prevent this incident.
That deep understanding is exactly what was required. If you don't understand how Mongo works, you don't understand the ways it can fail to do what you think it will do.
> If you're aware of them, you can trigger the thought process of "this query is pretty slow, maybe I should look into adding an index because I know that speeds things up".
So why not add an index for everything ahead of time, then every query will be fast, right?
Or maybe you need to have a sense of the cost of adding an index, then. Which may require knowing the different kinds of indexes and what the speed/cost tradeoffs are.
I think in all these cases, not every developer has to understand these things, but if you don't have anyone in your organization that understands them and is thinking about them, things can blow up for you at unexpected times that make your customers very unhappy. And then you might need to hire someone who understands them to fix the mess.
This is going to require being conversant with the terms in that ACID acronym.
> A deep understanding of how MongoDB works probably wasn't required to prevent this incident.
That deep understanding is exactly what was required. If you don't understand how Mongo works, you don't understand the ways it can fail to do what you think it will do.
> If you're aware of them, you can trigger the thought process of "this query is pretty slow, maybe I should look into adding an index because I know that speeds things up".
So why not add an index for everything ahead of time, then every query will be fast, right?
Or maybe you need to have a sense of the cost of adding an index, then. Which may require knowing the different kinds of indexes and what the speed/cost tradeoffs are.
I think in all these cases, not every developer has to understand these things, but if you don't have anyone in your organization that understands them and is thinking about them, things can blow up for you at unexpected times that make your customers very unhappy. And then you might need to hire someone who understands them to fix the mess.
Well, the interviewer did not complain about getting answer partially wrong, they complained about getting a blank stare back. The interviews are not like school exams, they are not graded by the number of exactly correct answer.
I don't remember what each letter of acronym means either. But if I was interviewee in this situation, I'd just start talking about transactions from user perspective -- how they either completeley pass or completely fail, and how even with power failure, you are not going to see partial data.
If I were interviewer, I'd accept this answer. No one cares about hearing exact words (0), "user will never see only a part of transaction" is good enough even if person never said the word "atomicity"
(0) According to some sources, Google runs scripted interviews where interviewer is non-technical, and grades answers against a checklist.
I don't remember what each letter of acronym means either. But if I was interviewee in this situation, I'd just start talking about transactions from user perspective -- how they either completeley pass or completely fail, and how even with power failure, you are not going to see partial data.
If I were interviewer, I'd accept this answer. No one cares about hearing exact words (0), "user will never see only a part of transaction" is good enough even if person never said the word "atomicity"
(0) According to some sources, Google runs scripted interviews where interviewer is non-technical, and grades answers against a checklist.
Anecdotal evidence: I didn't know what ACID meant, but I know each of the things:
• the transaction is the minimal (atomic) unit, and must never be split up;
• all transactions (and everything else too) must map valid states to valid states;
• parallel database transactions should be viewed as a directed asynchronous graph over valid states (like Git)
• store stuff on the hard drive rather than RAM (i.e. the hard part of making databases; filesystems are horrible!)
I have fairly deep understanding of these things; I just didn't know the cool acronym mnemonic. (Database engine devs don't tend to use it in their blog posts about the internals of SQLite or Postgres.) “If they didn't learn how I learnt, they don't know anything” is one of my pet peeves.
• the transaction is the minimal (atomic) unit, and must never be split up;
• all transactions (and everything else too) must map valid states to valid states;
• parallel database transactions should be viewed as a directed asynchronous graph over valid states (like Git)
• store stuff on the hard drive rather than RAM (i.e. the hard part of making databases; filesystems are horrible!)
I have fairly deep understanding of these things; I just didn't know the cool acronym mnemonic. (Database engine devs don't tend to use it in their blog posts about the internals of SQLite or Postgres.) “If they didn't learn how I learnt, they don't know anything” is one of my pet peeves.
To some degree the more competent the person in using ACID databases, the less likely they are to remember what 4 random words are hidden behind ACID. If someone uses a database that makes ACID guarantees there is no benefit to remembering what they are. It is just how the database works.
People only need to know about ACID if they are arguing about whether to use a database that doesn't implement it, or switching between databases some of which do and some of which don't provide strong ACID guarantees.
People only need to know about ACID if they are arguing about whether to use a database that doesn't implement it, or switching between databases some of which do and some of which don't provide strong ACID guarantees.
I disagree entirely with this, ACID only makes guarantees at the database level itself not the system as a whole; it's possible that using ACID in the database could render the system as a whole more vulnerable than not using it. Take for example MySQL/InnoDB which offers durability (the D in ACID). If you have a write heavy database you could end up in a situation where your server ends up stalling waiting for MySQL to commit the write to disk. In effect, your server ends up building a large buffer of queries to commit to MySQL and sits there waiting for MySQL to become available. If your server or system crashes, all of that data gets lost.
You may have been much better off, both performance and correctness, to relax MySQL's durability from flushing on every write, to flushing/committing every 1 second (using innodb_flush_log_at_trx_commit). Doing so improves performance by multiple orders of magnitude (it can be the order of 10-100x) to the point that your server is no longer sitting there queuing up queries for writes to complete. The database loses some guarantees about durability but the system as a whole gains better durability.
I could make similar arguments about the other properties as well, the point is that a competent person should actually know this stuff, understand it, not just memorize an acronym and boil the entire issue down to whether a database has ACID or not. It's not a binary matter, it's more of a continuum where each property can be tuned to various degrees. MySQL and Postgres both offer many options to tune the various ACID guarantees from having complete conformance to relaxing them with various upper bounds. Relaxing guarantees about a local property of the system can improve the guarantees of system as a whole.
You may have been much better off, both performance and correctness, to relax MySQL's durability from flushing on every write, to flushing/committing every 1 second (using innodb_flush_log_at_trx_commit). Doing so improves performance by multiple orders of magnitude (it can be the order of 10-100x) to the point that your server is no longer sitting there queuing up queries for writes to complete. The database loses some guarantees about durability but the system as a whole gains better durability.
I could make similar arguments about the other properties as well, the point is that a competent person should actually know this stuff, understand it, not just memorize an acronym and boil the entire issue down to whether a database has ACID or not. It's not a binary matter, it's more of a continuum where each property can be tuned to various degrees. MySQL and Postgres both offer many options to tune the various ACID guarantees from having complete conformance to relaxing them with various upper bounds. Relaxing guarantees about a local property of the system can improve the guarantees of system as a whole.
You're right but you or more precisely that interviewer is also wrong.
Understanding the way your storage works is crucial, yes. But you ensure people understand it not by asking "Explain ACID to me" but by asking to design a certain system and observing them figuring out requirements and asking questions. You then ask follow up question about consistency and reliability, you probably even can ask about benefits of using transactions but I wouldn't because there's plenty of databases that can be used reliably that don't have transactions.
Life is more complex than ACID, that's it. Investigate if person understands life, not that they were able to memorise bunch of magic acronyms.
For the life of me I don't remember what SOLID is and I totally fail to parse any written explanation of liskov substitution principle but whenever I see code that violates those I immediately go "please don't do this, that'll cause problems in the future".
Understanding the way your storage works is crucial, yes. But you ensure people understand it not by asking "Explain ACID to me" but by asking to design a certain system and observing them figuring out requirements and asking questions. You then ask follow up question about consistency and reliability, you probably even can ask about benefits of using transactions but I wouldn't because there's plenty of databases that can be used reliably that don't have transactions.
Life is more complex than ACID, that's it. Investigate if person understands life, not that they were able to memorise bunch of magic acronyms.
For the life of me I don't remember what SOLID is and I totally fail to parse any written explanation of liskov substitution principle but whenever I see code that violates those I immediately go "please don't do this, that'll cause problems in the future".
> Understanding the way your storage works is crucial, yes. But you ensure people understand it not by asking "Explain ACID to me" but by asking to design a certain system and observing them figuring out requirements and asking questions. You then ask follow up question about consistency and reliability, you probably even can ask about benefits of using transactions but I wouldn't because there's plenty of databases that can be used reliably that don't have transactions.
Who has time for that? Interviewers design interview for themselves, not for the interviewees. It's basically a trivia game at this point of time. A prime example of "Everything is obvious once you know the answer" syndrome.
Who has time for that? Interviewers design interview for themselves, not for the interviewees. It's basically a trivia game at this point of time. A prime example of "Everything is obvious once you know the answer" syndrome.
Now that is up to the skill of an interviewer how will they approach it.
Often you're time-limited and can't do actual code testing, and acronyms and buzzwords can be useful to do the initial filtering quickly. Of course, not expecting for candidates to reproduce what each letter means, but to show they've heard of it and have some idea what it's about.
> For the life of me I don't remember what SOLID is and I totally fail to parse any written explanation of liskov substitution principle but whenever I see code that violates those I immediately go "please don't do this, that'll cause problems in the future".
In my book this is a perfectly valid answer to that question, actually far better than just citing the book definition blindly. Interviews are all about trying to read between the lines, and often the whole impression of competence that candidate leaves to you is far more important than the actual answers.
Often you're time-limited and can't do actual code testing, and acronyms and buzzwords can be useful to do the initial filtering quickly. Of course, not expecting for candidates to reproduce what each letter means, but to show they've heard of it and have some idea what it's about.
> For the life of me I don't remember what SOLID is and I totally fail to parse any written explanation of liskov substitution principle but whenever I see code that violates those I immediately go "please don't do this, that'll cause problems in the future".
In my book this is a perfectly valid answer to that question, actually far better than just citing the book definition blindly. Interviews are all about trying to read between the lines, and often the whole impression of competence that candidate leaves to you is far more important than the actual answers.
> I totally fail to parse any written explanation of liskov substitution principle
Off the top of my head, "all child classes should be usable interchangeably when dealing with an interface/base class". I guess that's too imprecise to qualify as "written explanation"?
Off the top of my head, "all child classes should be usable interchangeably when dealing with an interface/base class". I guess that's too imprecise to qualify as "written explanation"?
This doesn’t sound correct to me. The original statement is, Let ϕ(x) be a property provable about objects x of type T. Then ϕ(y) should be true for objects y of type S where S is a subtype of T. Here, objects of a base class are replaced with objects of derived classes.
> This doesn’t sound correct to me.
Yet the GP says exactly the same thing as you said. He just ignored that the property belongs to the base class.
Yet the GP says exactly the same thing as you said. He just ignored that the property belongs to the base class.
The GP said "usable interchangeably", which is vague enough to admit a wide variety of interpretations, including the more rigorous one Koshkin posted.
Yet he got unambiguously declared wrong...
Or 'is a' implies 'behaves as a'.
Exactly, the acronyms should be viewed as tools for the interviewer, not as trivia for the interviewee to know. Asking the interviewee to explain ACID or SOLID is fine, but if they don't know the acronyms that shouldn't be an instant fail, it just means the interviewer has to put in a little more effort. The interviewer should go deeper and say "Given we're using this data to do this, which of the options of the MongoDB insert should we use?" or for SOLID, show them a piece of code and ask "What criticism do you have for this code? How would you refactor it?".
Given how some people study for these interviews, you might be better off going deep either way, to avoid getting rehearsed answers on these trivia questions.
Given how some people study for these interviews, you might be better off going deep either way, to avoid getting rehearsed answers on these trivia questions.
Thank you, ivanhoe!
When I read that sentence, I gasped and thought the same thing.
I've spent a career writing thousands of lines of application code that would have never been needed if the database had been designed properly. But it wasn't because the deployer didn't think they needed "deep knowledge" that day.
We stand on the shoulders of giants who figured out the fundamentals just for us.
The worse the violation (different data types in the same column, ugh!), the more application code (10X) will be needed.
The more code written, the more more maintenance (another 10X) needed.
And none of it ever gets replaced, just migrated to something else years later (if we're still in business).
ASIDE: As for the interview question, I would never ask what "ACID" meant. I'm more interested in understanding than recall. I would just give the candidate a poorly designed database and ask what they would do to fix it.
When I read that sentence, I gasped and thought the same thing.
I've spent a career writing thousands of lines of application code that would have never been needed if the database had been designed properly. But it wasn't because the deployer didn't think they needed "deep knowledge" that day.
We stand on the shoulders of giants who figured out the fundamentals just for us.
The worse the violation (different data types in the same column, ugh!), the more application code (10X) will be needed.
The more code written, the more more maintenance (another 10X) needed.
And none of it ever gets replaced, just migrated to something else years later (if we're still in business).
ASIDE: As for the interview question, I would never ask what "ACID" meant. I'm more interested in understanding than recall. I would just give the candidate a poorly designed database and ask what they would do to fix it.
And that's why you have senior engineers and junior ones.
If you are trying to hire, expecting every junior dev to know all this stuff means you'll skip a lot of candidates who can contribute a lot to your business.
They shouldn't be trusted with writing things from scratch. But if your codebase is mature enough, getting db connections, making endpoints atomic and stuff like that already should have a convention so your junior engineer wouldn't have to do it on their own.
If you are trying to hire, expecting every junior dev to know all this stuff means you'll skip a lot of candidates who can contribute a lot to your business.
They shouldn't be trusted with writing things from scratch. But if your codebase is mature enough, getting db connections, making endpoints atomic and stuff like that already should have a convention so your junior engineer wouldn't have to do it on their own.
> I've spent a career writing thousands of lines of application code that would have never been needed if the database had been designed properly.
And then there's the time that I rewrote a small database application, but couldn't really reduce the byzantine schema, as that's just how our business works. However, I did reduce the main query from 727 lines to 93, and made it run twice as fast. And then, of course, the project was canceled because of politics, and the group still struggles with their terrible application, 3 years later.
I should really write a book about terrible management in Fortune 250's.
And then there's the time that I rewrote a small database application, but couldn't really reduce the byzantine schema, as that's just how our business works. However, I did reduce the main query from 727 lines to 93, and made it run twice as fast. And then, of course, the project was canceled because of politics, and the group still struggles with their terrible application, 3 years later.
I should really write a book about terrible management in Fortune 250's.
[deleted]
I think your comment is basically just reacting to the title or a snippet of the article and not really giving the OP due respect. For example transactions are clearly a part of the api for a SQL database so not having deep knowledge and not using transactions are not equivalent things.
I think the article also touches on the fact that when you look up the definition of ACID you get a rough description of the acronym and then a description of how it is mostly a marketing term and all the ways real systems can deviate from it. So really the deep knowledge is not so much what ACID means as what it doesn’t.
The article is basically complaining that knowing what ACID stands for is not really a fundamental and that knowing how to use a database is. So I don’t think you really disagree with the OP much either.
I think the article also touches on the fact that when you look up the definition of ACID you get a rough description of the acronym and then a description of how it is mostly a marketing term and all the ways real systems can deviate from it. So really the deep knowledge is not so much what ACID means as what it doesn’t.
The article is basically complaining that knowing what ACID stands for is not really a fundamental and that knowing how to use a database is. So I don’t think you really disagree with the OP much either.
I don't think that transactions count as deep database knowledge. Those are the absolute fundamentals that most developers need on a daily basis and not some dusty computer science stuff. If you have never heard of them it is fair to say that you should take a step back and learn the basics of your profession before being allowed to touch a production database. But the thing is of course - you don't know what you don't know.
Great point:
> you don't know what you don't know
aka knowledge is what you have after you forgot everything you learned.
> you don't know what you don't know
aka knowledge is what you have after you forgot everything you learned.
One of projects I was called to help some time ago set up a service with over a thousand servers. They reached out to help improve the performance because in their opinion the network and network filesystem were too slow.
They invested incredible amount of time in learning technology and scaling their application but forgot about need to learn fundamentals -- data structures and efficiency.
Fast forward 1,5 years, the application as I left it ran on a single server using about 10% of its capacity. A second server is just a hot standby backup.
The service went literally from being able to process tens of transactions to a hundred thousand transactions per second on a single node.
More than that, we threw away most of the exotic technology that was used there -- greatly improving team productivity. The implementation is simpler than ever with layers upon layers of microservices replaced with regular method calls and a lot of infrastructure basically removed without a need to replace it with anything.
People who do not learn from history (fundamentals) are doomed to repeat the same mistakes.
They invested incredible amount of time in learning technology and scaling their application but forgot about need to learn fundamentals -- data structures and efficiency.
Fast forward 1,5 years, the application as I left it ran on a single server using about 10% of its capacity. A second server is just a hot standby backup.
The service went literally from being able to process tens of transactions to a hundred thousand transactions per second on a single node.
More than that, we threw away most of the exotic technology that was used there -- greatly improving team productivity. The implementation is simpler than ever with layers upon layers of microservices replaced with regular method calls and a lot of infrastructure basically removed without a need to replace it with anything.
People who do not learn from history (fundamentals) are doomed to repeat the same mistakes.
Something people seem to forget when designing systems based on what they've learned is that modern individual machines have approximately the power of the top supercomputer ~20 years ago, and a few racks of modern machines can in many ways match the top supercomputer of ~10 years ago. Approaches to solve large problems are continually changing and it's almost always worth a big-picture design session when looking at a familiar-seeming problem. By the time students graduate from a 4-year program computing is about 10X better than when they started.
I still remember some interviews within the last decade where single-core machines with a few GB of RAM were a default assumption for whiteboard designs, or that spinning disks were the default.
I still remember some interviews within the last decade where single-core machines with a few GB of RAM were a default assumption for whiteboard designs, or that spinning disks were the default.
Well, I catch myself that I am not "updating" the state of my understanding of hardware. Just recently I caught myself putting a lot of effort into solving a problem that only exists if persistent storage is too slow to be used for calculations. Then I facepalmed myself hard when I realized that I can just move that entire 500GB data structure to an NVMe and treat it almost as if it was in memory.
But in general I think that the problem isn't that people are not "updating" their understanding. Even 10 years ago it wasn't a huge problem getting hundreds of thousands of transactions per second on a single modest machine.
The problem rather is people relying on more and more layers of abstractions for vary small gains.
Example: I get that Python is a nice language (for somebody that does not know Lisp). But is it worth it to choose Python for a little bit improvement in productivity for a problem that requires a lot of throughput, to then suffer performance issues, to then spend many times more effort on trying to improve performance? I don't think so.
Or more in my space: is Spring Data (Java) worth the very incremental productivity improvements if it completely destroys your application performance? The application I described in my parent post used Spring Data MongoDB which kinda means it was fetching entities one by one which is extremely costly.
By replacing it with bare MongoDB reactive driver and ensuring data is being streamed in large batches (why read one user data if you can read 10k at a time) and getting rid of costly aggregations in favour of application side processing we have improved throughput by many orders of magnitude WHILE reducing load on MongoDB.
Granted, there is a little bit of additional complexity (on the order of 10% more of application code) but just the performance improvements mean that the team can breathe and focus on other problems like modelling the domain correctly.
But in general I think that the problem isn't that people are not "updating" their understanding. Even 10 years ago it wasn't a huge problem getting hundreds of thousands of transactions per second on a single modest machine.
The problem rather is people relying on more and more layers of abstractions for vary small gains.
Example: I get that Python is a nice language (for somebody that does not know Lisp). But is it worth it to choose Python for a little bit improvement in productivity for a problem that requires a lot of throughput, to then suffer performance issues, to then spend many times more effort on trying to improve performance? I don't think so.
Or more in my space: is Spring Data (Java) worth the very incremental productivity improvements if it completely destroys your application performance? The application I described in my parent post used Spring Data MongoDB which kinda means it was fetching entities one by one which is extremely costly.
By replacing it with bare MongoDB reactive driver and ensuring data is being streamed in large batches (why read one user data if you can read 10k at a time) and getting rid of costly aggregations in favour of application side processing we have improved throughput by many orders of magnitude WHILE reducing load on MongoDB.
Granted, there is a little bit of additional complexity (on the order of 10% more of application code) but just the performance improvements mean that the team can breathe and focus on other problems like modelling the domain correctly.
The author’s point still stands.
If you’re implementing interactions with a fairly new database (let’s face it. 10 years ago MongoDB was basically experimental) then it is indeed incumbent on you to understand in greater details what you’re doing.
But if you’re building a CRUD web app, that’s maybe only used internally by your company and backed by SQL Server you probably have to never give it a thought.
Either way, lack of knowledge of a particular acronym, one which is described as basically meaningless in texts that describe it, is not a deal breaker of any sort. And it doesn’t apply just to ACID but most acronyms in tech today, such as REST/SOLID, etc like the author points out.
If you’re implementing interactions with a fairly new database (let’s face it. 10 years ago MongoDB was basically experimental) then it is indeed incumbent on you to understand in greater details what you’re doing.
But if you’re building a CRUD web app, that’s maybe only used internally by your company and backed by SQL Server you probably have to never give it a thought.
Either way, lack of knowledge of a particular acronym, one which is described as basically meaningless in texts that describe it, is not a deal breaker of any sort. And it doesn’t apply just to ACID but most acronyms in tech today, such as REST/SOLID, etc like the author points out.
> But if you’re building a CRUD web app, that’s maybe only used internally by your company and backed by SQL Server you probably have to never give it a thought.
Until you get complaints that the hour-long reconciliation transaction you run in the morning is locking out all the users, or there's an API a customer hits that's erroring out 1% of the time because Sybcoughuh, Wonderful Microsoft SQL Server has detected a deadlock, and this is causing us to lose customer orders, or...
No, you really need to know what ACID means, and what tradeoffs are made to balance ACID with other considerations in your chosen database implementation. Lumping ACID in with vague heuristics like SOLID on the basis that they're both acronyms is just plain ignorance.
Until you get complaints that the hour-long reconciliation transaction you run in the morning is locking out all the users, or there's an API a customer hits that's erroring out 1% of the time because Sybcoughuh, Wonderful Microsoft SQL Server has detected a deadlock, and this is causing us to lose customer orders, or...
No, you really need to know what ACID means, and what tradeoffs are made to balance ACID with other considerations in your chosen database implementation. Lumping ACID in with vague heuristics like SOLID on the basis that they're both acronyms is just plain ignorance.
I think it is possible that an engineer knows about the need to use transactions while doing certain kinds of DB updates while not knowing the exact definition of the ACID acronym.
I suspect the author of this article would have no objections to probing if a candidate understands transactions and indices.
I suspect the author of this article would have no objections to probing if a candidate understands transactions and indices.
Exactly, I've spent years optimizing RDBMS and columnar databases and I couldn't at this moment fully explain the definition of ACID to you without having a wee google refresher.
I mean, I learned ACID's definition, nodded because it made sense, realised that MySQL's default (at the time) MyISAM storage engine wasn't ACID, it flat out failed at C, so encouraged my LAMP stack using friends to switch to InnoDB, then switched to Postgres myself.
Likewise, I use a lot of distributed systems, but I don't think in terms of the CAP theorem on the regular.
Rather I just think about all the fun ways each distributed system can die in, and while ZK is CP (although I'm dubious about the P), and Kafka is CA, the failure mode is roughly the same - you lose enough cluster members and shit breaks.
I know that my k-safety (k being how many nodes I can lose and still have a system) for ZK is N - (N +1)/2 (assuming truncating int division), and as for Kafka, well... it really depends on the failure.
But the usual conservative design for a Kafka cluster is n * 3 replicas on a 3 AZ stretch cluster with min.insync.replicas set to num.replicas - 1,leading to a k safety of 1.
I'm a fan of the 2.5 stretch cluster, if I have to do a stretch cluster, I generally prefer separate clusters replicating.
So for a 2.5, you'd go 2 AZs with M brokers and N ZK nodes, where M % 2 == 0 && N % 2 == 0, with one ZK in the third AZ as the tiebreaker.
You can now lose an entire AZ, and still have a quorum, while paying 33% less inter-AZ traffic from your Kafka clients.
Basically, theory underpins all my work, but the theory and the reality are quite different things.
I mean, I learned ACID's definition, nodded because it made sense, realised that MySQL's default (at the time) MyISAM storage engine wasn't ACID, it flat out failed at C, so encouraged my LAMP stack using friends to switch to InnoDB, then switched to Postgres myself.
Likewise, I use a lot of distributed systems, but I don't think in terms of the CAP theorem on the regular.
Rather I just think about all the fun ways each distributed system can die in, and while ZK is CP (although I'm dubious about the P), and Kafka is CA, the failure mode is roughly the same - you lose enough cluster members and shit breaks.
I know that my k-safety (k being how many nodes I can lose and still have a system) for ZK is N - (N +1)/2 (assuming truncating int division), and as for Kafka, well... it really depends on the failure.
But the usual conservative design for a Kafka cluster is n * 3 replicas on a 3 AZ stretch cluster with min.insync.replicas set to num.replicas - 1,leading to a k safety of 1.
I'm a fan of the 2.5 stretch cluster, if I have to do a stretch cluster, I generally prefer separate clusters replicating.
So for a 2.5, you'd go 2 AZs with M brokers and N ZK nodes, where M % 2 == 0 && N % 2 == 0, with one ZK in the third AZ as the tiebreaker.
You can now lose an entire AZ, and still have a quorum, while paying 33% less inter-AZ traffic from your Kafka clients.
Basically, theory underpins all my work, but the theory and the reality are quite different things.
>Because of such attitude of the previous dev team my client ended up with DB integrity ruined
Same thing here, our legacy codebase avoided using transactions, and to this day, after migrating to a new codebase, we still get reports from clients here and there about errors which stem from inconsistent data
Same thing here, our legacy codebase avoided using transactions, and to this day, after migrating to a new codebase, we still get reports from clients here and there about errors which stem from inconsistent data
Damn, and I've always felt like I must have some kind of OCD whenever I'm getting a refresher on MVCC/MGA isolation levels and their guarantees and the associated auxiliary locking (say, select for update with lock in Firebird, for example) just to be sure I'm not writing something stupid. Simply not using transactions at all is something I can't even fathom. That's basically no better than blindly mutating shared in-memory data structures in a multi-threaded program. Who would write something like this?
> Who would write something like this?
Many, many people.
When I read it, my first thought was that "an API" in this case was SQL.
Understanding why we should use database transactions is not the same, to my mind, as understanding database internals. When the author said they can treat the database as a "black box", my thinking was the they didn't necessarily need knowledge of the database's implementation. People can understand how to effectively work with a database in the general sense (i.e. use transactions) without a deep knowledge of every implementation.
When we start talking about MongoDB, I think that's really interesting because it is so different from our traditional, SQL-first databases like PostgreSQL, Oracle, etc. Our understanding of SQL-first databases isn't all that useful when we try to apply it to MongoDB.
At that point I think we are in agreement: it's unreasonable to think we can treat a MongoDB database as a black box if our base knowledge is only SQL and SQL-first databases. We probably do need someone who has already dug in and dealt with the implementation, how MongoDB really gets things done.
Understanding why we should use database transactions is not the same, to my mind, as understanding database internals. When the author said they can treat the database as a "black box", my thinking was the they didn't necessarily need knowledge of the database's implementation. People can understand how to effectively work with a database in the general sense (i.e. use transactions) without a deep knowledge of every implementation.
When we start talking about MongoDB, I think that's really interesting because it is so different from our traditional, SQL-first databases like PostgreSQL, Oracle, etc. Our understanding of SQL-first databases isn't all that useful when we try to apply it to MongoDB.
At that point I think we are in agreement: it's unreasonable to think we can treat a MongoDB database as a black box if our base knowledge is only SQL and SQL-first databases. We probably do need someone who has already dug in and dealt with the implementation, how MongoDB really gets things done.
> Because of such attitude of the previous dev team my client ended up with DB integrity ruined. Previous guys somehow didn't know they should use transactions when updating/deleting stuff in the DB, because hey, it's just API you call, who cares of mambo-jumbo happening behind the curtains, right?
There's a difference between knowing the internal implementation, and the pros/cons of different usage patterns. What you're talking about is having mastery over an interface. This is distinct from knowing the internal implementation.
As someone who used to work as a hardware engineer, I can assure you that there's a tremendous amount of hardware internal-implementation-details that you're ignorant of. But that doesn't get in the way of you being you a great developer, as long as you spend some time understanding the interface and the recommended ways to use the hardware black-box. The same applies to what the OP is describing.
There's a difference between knowing the internal implementation, and the pros/cons of different usage patterns. What you're talking about is having mastery over an interface. This is distinct from knowing the internal implementation.
As someone who used to work as a hardware engineer, I can assure you that there's a tremendous amount of hardware internal-implementation-details that you're ignorant of. But that doesn't get in the way of you being you a great developer, as long as you spend some time understanding the interface and the recommended ways to use the hardware black-box. The same applies to what the OP is describing.
Came here to address his part of the article.
My statement is that it was a terrible example, but the core principle stands. Yes, we should all definitely understand how the storage layer works (and anything else that could corrupt data) but no one needs to have the laundry list of acronyms memorized.
My statement is that it was a terrible example, but the core principle stands. Yes, we should all definitely understand how the storage layer works (and anything else that could corrupt data) but no one needs to have the laundry list of acronyms memorized.
> Previous guys somehow didn't know they should use transactions when updating/deleting stuff in the DB, because hey, it's just API you call, who cares of mambo-jumbo happening behind the curtains, right?
Sure, you're right, but I'd argue that transactions are is hardly "database internals". Transactions are day 2 of "using a database 101".
You don't have to know what ACID means or how transactions are implemented to use them correctly.
I'd rather hire a candidate that knows how to use transactions but has no idea what ACID is than one who knows what ACID means and how it's implemented but doesn't know when or why they should use transactions in real life. (Of course the second case is rare, but this is just an example).
Sure, you're right, but I'd argue that transactions are is hardly "database internals". Transactions are day 2 of "using a database 101".
You don't have to know what ACID means or how transactions are implemented to use them correctly.
I'd rather hire a candidate that knows how to use transactions but has no idea what ACID is than one who knows what ACID means and how it's implemented but doesn't know when or why they should use transactions in real life. (Of course the second case is rare, but this is just an example).
I get the feeling that many people do not even know what problems databases solve. It seems to be common to treat them as an extension to the application's memory that somehow isn't lost between restarts, and the properties, operational caveats and other capabilities of the database are not even considered.
So true. I see a lot backend candidates these days who learned only how to use the ORM of their framework of choice, plus migrations, and that's it.
I don't think use of transactions or any other query or modeling hygiene are "internals". You can just as easily consider SQL to be an API (or really a DSL) for the black box of DB internals. In most cases, you're better off not looking inside the box because DB vendors only guarantee behavior at the API level. You can always identify and improve bottlenecks if/when they actually impact application function. I can write a load of crummy queries, expose them in an API and then just apply some clever caching and the DB will just never be a bottleneck no matter how little I understand it.
> Previous guys somehow didn't know they should use transactions when updating/deleting stuff in the DB, because hey, it's just API you call, who cares of mambo-jumbo happening behind the curtains, right?
I would consider transactions part of the api. I use them all the time in postgres and I'd hardly call myself an expert in the low level details
I would consider transactions part of the api. I use them all the time in postgres and I'd hardly call myself an expert in the low level details
Exactly. Maybe not everyone needs to be an expert in databases and distributed systems, but if you’re developing web technologies, then someone needs to be the expert. They will need to have the vocabulary to communicate with the rest of the team.
Just to play devil's advocates, our juniors LOVE putting everything in a transaction and locking up the client's legacy database.
Putting althe modification resulting from an API call within a transaction doesn't do anything bY itself.
The author covers your perspective:
> But, that’s only because you are a noob and you don’t deal with the things that I have deal with.
> Someone on the internet
In conclusion: not all software problems require the same implementation.
I don’t run a DB for some deterministic applications I have that can regenerate the same outputs such I don’t need to store it all. I could, in theory, regenerate an entire data set with a seed. Maybe app developers are ignorant and lazy for not designing systems that do that!
What does your one story have to do with “software engineering” as a field?
Sounds like people are conflating problem solving and personal career demands.
> But, that’s only because you are a noob and you don’t deal with the things that I have deal with.
> Someone on the internet
In conclusion: not all software problems require the same implementation.
I don’t run a DB for some deterministic applications I have that can regenerate the same outputs such I don’t need to store it all. I could, in theory, regenerate an entire data set with a seed. Maybe app developers are ignorant and lazy for not designing systems that do that!
What does your one story have to do with “software engineering” as a field?
Sounds like people are conflating problem solving and personal career demands.
I've found bad interview questions fall into one of three camps:
1. The interviewers aren't subject matter experts themselves but are hiring managers, so they've taken their questions from someone else.
2. The company gets far more good candidates apply than they can hire, so the process is less about removing bad candidates and more about having some arbitrary way of whittling down good candidates.
3. The interviewer is trying to prove how clever they are or the company is. Sometimes for their own ego but more often because they think it might make the company more attractive to work at.
If there is a theoretical question I'd normally Google / DDG instead of memorise then I'd usually be honest in the interview and say "it's something like xyz but in truth this is something I double check these days as, pragmatically speaking, it is not a problem that comes up frequently. However it is something I have done before [cite example]".
If the questions is asked because of reason 1 or 3 from above, then usually they're happy with that answer. If they ask it because of 2, well as much as I might disagree with their interview process, I'm still not the candidate they're looking for. So we still part of good terms.
1. The interviewers aren't subject matter experts themselves but are hiring managers, so they've taken their questions from someone else.
2. The company gets far more good candidates apply than they can hire, so the process is less about removing bad candidates and more about having some arbitrary way of whittling down good candidates.
3. The interviewer is trying to prove how clever they are or the company is. Sometimes for their own ego but more often because they think it might make the company more attractive to work at.
If there is a theoretical question I'd normally Google / DDG instead of memorise then I'd usually be honest in the interview and say "it's something like xyz but in truth this is something I double check these days as, pragmatically speaking, it is not a problem that comes up frequently. However it is something I have done before [cite example]".
If the questions is asked because of reason 1 or 3 from above, then usually they're happy with that answer. If they ask it because of 2, well as much as I might disagree with their interview process, I'm still not the candidate they're looking for. So we still part of good terms.
In an interview, they gave me access to their codebase and asked me to fix an actual (small) bug. The interviewer showed me where the codebase was, how to replicate the bug, all of that took less than 10 mins. Then he left, told me to give a shout once I am done. Took me about half hour to do it, another 10 mins to explain what I did to fix the bug.
There were no other questions - no white board stuff, no algorithm questions, no stupid questions like "where do you see yourself in 5 years?" etc. I still remember the interview nearly a decade later.
Obviously this won't work for all jobs, but this will work surprisingly well for a large number of jobs, because most jobs are ordinary. One doesn't need to know complex algorithms to do these
There were no other questions - no white board stuff, no algorithm questions, no stupid questions like "where do you see yourself in 5 years?" etc. I still remember the interview nearly a decade later.
Obviously this won't work for all jobs, but this will work surprisingly well for a large number of jobs, because most jobs are ordinary. One doesn't need to know complex algorithms to do these
My entire "interview" at Netflix was me helping their top database people solve a severe performance problem in their operational systems. In the two hours I was there, I diagnosed the root cause and showed them how to fix it doing something they did not know was possible. Best and easiest job interview I've ever had.
They tested it after I left (a >100x improvement apparently) and called me the next day to ask when I could start.
They tested it after I left (a >100x improvement apparently) and called me the next day to ask when I could start.
How did you get into that position tho?
I did something similar and loved it but a friend pointed out That he will fall over with an anxiety attack in such a situation. I always thought the best case would be to offer a choice to the candidate among 2 or 3 different ways to do the interview.
>I did something similar and loved it but a friend pointed out That he will fall over with an anxiety attack in such a situation.
I might, too, but I frankly might do the same to an even more extreme degree for a particularly challenging real-time 1-on-1 interview question from someone staring at me.
All things considered, I think I'd prefer being left alone in a room where I don't feel them over my shoulder while constantly wondering what they might be thinking with each passing second. I can take a few minutes to relax a little, look over the code, and try to figure it out.
I might, too, but I frankly might do the same to an even more extreme degree for a particularly challenging real-time 1-on-1 interview question from someone staring at me.
All things considered, I think I'd prefer being left alone in a room where I don't feel them over my shoulder while constantly wondering what they might be thinking with each passing second. I can take a few minutes to relax a little, look over the code, and try to figure it out.
I don't understand why this would give someone anxiety attack - it would suck if the interviewer sat next to me and watched my every keystroke, it wasn't that in my case. He just set me up and left to do his job. I was on my own, until I finished and called him over to show the bugfix.
I have anxiety issues when an interviewer asks me some obscure syntax option for grep command or some b-tree stuff that I learned in college decade ago (and never used in real life).
I have anxiety issues when an interviewer asks me some obscure syntax option for grep command or some b-tree stuff that I learned in college decade ago (and never used in real life).
I don't understand why that grep stuff would give you anxiety. It is a dumb trivia question, just say you don't know.
But I don't understand because I am not you. Different things give different people anxiety. Anxiety isn't something you reason your way into or out of.
But I don't understand because I am not you. Different things give different people anxiety. Anxiety isn't something you reason your way into or out of.
> I don't understand why this would give someone anxiety attack
What can you do to understand why this happens?
What can you do to understand why this happens?
And it rises legal issues in several countries
What legal issues?
I'm not OP, but I imagine the concern is doing work for the company (i.e. fixing a bug in their codebase) without getting paid at least minimum wage for it.
Ah, I assumed it was just a realistic looking exercise. If they actually used code written by the applicant, I can definitely see all kinds of problems!
If I were designing this question I would probably use a real bug that we recently fixed. We'd already have a sense for how hard of a problem it was, and there's no worry about unpaid work.
Adding to that list:
4. Interviewer did not have time set aside to prepare for the interview, and sort of has to wing it. I was surprised to realize how often this happens.
4. Interviewer did not have time set aside to prepare for the interview, and sort of has to wing it. I was surprised to realize how often this happens.
> “Yes, every programmer should know how their hardware works as one cannot write an efficient software without that knowledge 4, and I wouldn’t be able to do my job if I didn’t know anything about computer internals, yak, yak, yak…”
I agree that you don't need to know anything about von Neumann Architecture and the likes in order to be a Web Dev.
But I wouldn't want to hire someone who doesn't have basic knowledge about data structures and runtime complexity. It's just too easy to have something work fine in development, but choke in production due to accidentally quadratic behaviour.
I agree that you don't need to know anything about von Neumann Architecture and the likes in order to be a Web Dev.
But I wouldn't want to hire someone who doesn't have basic knowledge about data structures and runtime complexity. It's just too easy to have something work fine in development, but choke in production due to accidentally quadratic behaviour.
Usually I had funny but opposite experience when doing consulting for performance optimizations.
At some points either teams converged to people without basic knowledge of data structures or they rushed with something that works (when N in O(N) was small) then scaled up. They get used to lower performance slowly like boiled frog.
Then someone is tasked with optimization, but instead of attacking data structure at hand, usually those people tried to optimize pieces of the puzzle for smaller gain, then they ended up with small hacks to shave some execution time, with unmaintainable code which lead to serious bugs.
At some points either teams converged to people without basic knowledge of data structures or they rushed with something that works (when N in O(N) was small) then scaled up. They get used to lower performance slowly like boiled frog.
Then someone is tasked with optimization, but instead of attacking data structure at hand, usually those people tried to optimize pieces of the puzzle for smaller gain, then they ended up with small hacks to shave some execution time, with unmaintainable code which lead to serious bugs.
This is the kind of observation that should be central to CS, but isn't.
While academic algorithm astronauts write papers which include Greek math symbols and can be proven correct, most people do... something else.
Optimising the something else would be a huge benefit to the industry. But no one even looks at it. It's just a thing that happens, and there's no research into fixing it or making it better.
While academic algorithm astronauts write papers which include Greek math symbols and can be proven correct, most people do... something else.
Optimising the something else would be a huge benefit to the industry. But no one even looks at it. It's just a thing that happens, and there's no research into fixing it or making it better.
I feel like this is moving in the opposite direction! I think the issue is that most of the time the "something else" people write with poor algorithms is actually an application of the correct, efficient algorithms from formal papers written by academics, but they don't know it and thus don't use it.
I think the main thing we want to avoid here is situations like the following:
- interviewer asks candidate to determine the complexity of some algorithms
- candidate is nervous, has a minor brain-fart and flubs a couple of them
- interviewer evaluates candidate as lacking basic CS knowledge you learn in 1st year of any CS course
Ok in this case we're talking about big-O, but it could really be anything.
- interviewer asks candidate to determine the complexity of some algorithms
- candidate is nervous, has a minor brain-fart and flubs a couple of them
- interviewer evaluates candidate as lacking basic CS knowledge you learn in 1st year of any CS course
Ok in this case we're talking about big-O, but it could really be anything.
Just say everything is O(2^N) and you’ll be technically correct most of the time. If the interviewer says that it’s wrong, they don’t know their fundamentals.
Honestly, the N+1 query problem with ORMs is more important for most web developers than the data structures used and their complexity.
Can you explain? I clearly forgot the fundamentals :p
(Taking a Algorithms course now, so I might get it wrong, if so, RIP to my finals in a few weeks)
I think O notation represents the upper bound, so a complexity of N can also be said as O(N^2) or O(N^3), but that isn't particularly useful.
In other words, if f(x)/g(x) < inf when x->inf, then f(x) = O(g(x)).
Then there's omega-notation (lower bound, f(x)/g(x) > 0) and theta-notation (both lower and upper bound, 0 < f(x)/g(x) < inf).
I think O notation represents the upper bound, so a complexity of N can also be said as O(N^2) or O(N^3), but that isn't particularly useful.
In other words, if f(x)/g(x) < inf when x->inf, then f(x) = O(g(x)).
Then there's omega-notation (lower bound, f(x)/g(x) > 0) and theta-notation (both lower and upper bound, 0 < f(x)/g(x) < inf).
Yeah this is right. Big O can be thought of as meaning “eventually always smaller than a multiple of” so f(n) = O(g(n)) as n -> inf means that there is some N (the “eventually” part) and M (the “multiple of” part) such that, for every n > N, |f(n)| < M g(n).
I had a good chuckle at this, +1 !
I think it should depend on how much money you are intending to pay the candidate and how desperate your company is to hire.
By my experience, some companies need a competent body in the seat ASAP, and some are looking for world-class engineers.
By my experience, some companies need a competent body in the seat ASAP, and some are looking for world-class engineers.
That's true to some extent, but I am quite sure even the worldest-of-class engineers have tripped up on a question or two and went on to get a rejection. If such people truly are talented then they'll have no trouble getting hired elsewhere and the companies in question would be blissfully unaware what they missed out on.
Such companies deserve to fail if so, as the supply of such engineers is limited and allegedly they need such engineers to survive :)
Think Sears vs Amazon-- which company would know what gold was if they fell into a pile of it?
Think Sears vs Amazon-- which company would know what gold was if they fell into a pile of it?
I am not sure the failure of any company has ever hinged on the hiring decisions of a few developers. If you take a look into the downfall of Sears you'll see long-term strategic missteps that aren't related to recruitment in their software engineering teams.
Also I don't think a company deserves to fail if they've got a bit too much faith in a slightly flawed recruitment process. The entire point of us sharing posts like this on here and discussing them is to learn and even help each other, isn't it?
Also I don't think a company deserves to fail if they've got a bit too much faith in a slightly flawed recruitment process. The entire point of us sharing posts like this on here and discussing them is to learn and even help each other, isn't it?
But if they’re not competent, then you’ll be stuck debugging/rewriting 6 months of the the incompetent engineer’s work
Completely agree and have been burned this way as well. If you need engineers but cant even find competent ones (something ive seen before), then the only choice is to put them on prototype work/non-critical projects
Yeah I agree that there is no need to go overly academic in interviews. I wouldn't ask about the different O-Notations and what they mean.
But if I give a code sample that has three nested for loops over some array and doing string comparisons, the candidate should at least have a vague idea that this might be problematic and could better be solved using sets or maps.
But if I give a code sample that has three nested for loops over some array and doing string comparisons, the candidate should at least have a vague idea that this might be problematic and could better be solved using sets or maps.
Yeah I agree that it could be better to explore this sort of stuff by looking at actual code. To be honest though you could come at it from more of an academic angle if the candidate has a more theoretical background. It's just about how the question is presented, being flexible with the response and ready to adapt if things don't go as planned. Maybe I'm kind of derailing the topic here to grind my own axe around some people's attitude towards recruitment :D
Not knowing how databases work is a pretty common issue I come across with colleagues. Even basic things like normal forms (which take about 5 minutes to understand), and what transactions are. I think it’s why non-RDBMS datastores are so popular, because they generally just ignore all the problems RDBMS are trying to solve, so the developer isn’t inconvenienced by thinking about them.
Non rdbms are popular because at scale everything that's nice about "classic" rdbms are what makes them ultra slow.
edit: the features I refer to are stuff like foreign keys, cascades, dynamic fields, complex joins, etc. Once your stuff is spread out you loose some of these by definition and if you're still sticking to a single DB with massive stores you're not getting much humpf out of it anymore.
I 100% agree there are way too many fools that think their DB is big and try non rdbms too early and waste too much time. But I still think their lack of recent popularity is due to their own limitations.
edit: the features I refer to are stuff like foreign keys, cascades, dynamic fields, complex joins, etc. Once your stuff is spread out you loose some of these by definition and if you're still sticking to a single DB with massive stores you're not getting much humpf out of it anymore.
I 100% agree there are way too many fools that think their DB is big and try non rdbms too early and waste too much time. But I still think their lack of recent popularity is due to their own limitations.
Not really, they remain fast enough, there’s just no easy way to scale them past a certain point. Or there wasn’t anyway, I believe with spanner and CockroachDB we’re getting pretty close.
But 99% of projects do not come even close to the scale where you need that.
But 99% of projects do not come even close to the scale where you need that.
There are very few RDBMS usecases where scale is a legitimate performance concern. Index scans scale to a very, very high cardinality.
There are use cases that aren’t best suited to an RDBMS at any scale, like managing a collection of denormalized documents, or a graph. But in my experience, considering these factors isn’t what’s motivating adoption of Mongo, or Dynamo. Your key lookups will always be fast (until you get a hot shard), but when you cram a highly relational set into a document db, eventually you’re forced to realize that there’s a lot of mutations and access patterns that you simply can’t implement, because you chose the wrong tool for the job.
Either that or you’ll decide that it’s a good idea to implement your own concurrency control mechanism on top of your document database. Which I’ve personally seen at least 3 people try and do.
There are use cases that aren’t best suited to an RDBMS at any scale, like managing a collection of denormalized documents, or a graph. But in my experience, considering these factors isn’t what’s motivating adoption of Mongo, or Dynamo. Your key lookups will always be fast (until you get a hot shard), but when you cram a highly relational set into a document db, eventually you’re forced to realize that there’s a lot of mutations and access patterns that you simply can’t implement, because you chose the wrong tool for the job.
Either that or you’ll decide that it’s a good idea to implement your own concurrency control mechanism on top of your document database. Which I’ve personally seen at least 3 people try and do.
Except they are also popular with people that are not working "at scale" where that would matter.
obligatory https://youtu.be/b2F-DItXtZs webscale database discussion
This promulgation and advocacy of laziness and know-nothing-ism is frightening and all too commonplace nowadays.
The ACID example was a bad one, granted, but everyone seems to be giving up after identifying that. I also think that this post is not about interviewing. I am much more concerned with the line the author takes about knowing how a digital computer work is even necessary. How have we arrived at this point?
As an example, how seriously would you take an aircraft flight controls software engineer who willfully chooses to never learn how an airplane flies? What if they just claimed that their sole responsibility was to take flight requirements from “people who know how airplanes work” and implement code? Knowing how horrific the results of this attitude is in even low pressure business situations, would you get on the plane he designed the software for?
People who live by and spread the idea that one only need know the things he or she is ‘required to know’ are dangerous for engineering organizations. When something goes horribly wrong, your mission critical system crashes, when you get dragged before the higher ups to explain the situation and fix it, the excuse ‘I don’t have to know how this works to do my job’ simply won’t fly.
People have lost all respect for the pursuit of excellence. Ask yourself who you would like to have on your team: someone who stops at learning the bare minimum to get paid and inevitably ruins the integrity of a product or someone who continually pursues getting better at their job and accelerates the people around them? Would you rather work with someone who values knowledge or someone who doesn’t?
I think it is drastically foolish to defend the toleration of those who aren’t even willing to learn basics like how a computer works in the name of inclusivity, utilitarianism, or egalitarianism.
Furthermore, I know a slew of extremely bright, tenacious, and committed young people who are ruthlessly pursuing excellence in engineering. If given the chance and enough time they will easily outperform, outwork, and surpass anyone who adheres to these lazy practices. If you do think this way and don’t want these kids to take your job then I suggest you buckle down a bit and learn some basics.
The ACID example was a bad one, granted, but everyone seems to be giving up after identifying that. I also think that this post is not about interviewing. I am much more concerned with the line the author takes about knowing how a digital computer work is even necessary. How have we arrived at this point?
As an example, how seriously would you take an aircraft flight controls software engineer who willfully chooses to never learn how an airplane flies? What if they just claimed that their sole responsibility was to take flight requirements from “people who know how airplanes work” and implement code? Knowing how horrific the results of this attitude is in even low pressure business situations, would you get on the plane he designed the software for?
People who live by and spread the idea that one only need know the things he or she is ‘required to know’ are dangerous for engineering organizations. When something goes horribly wrong, your mission critical system crashes, when you get dragged before the higher ups to explain the situation and fix it, the excuse ‘I don’t have to know how this works to do my job’ simply won’t fly.
People have lost all respect for the pursuit of excellence. Ask yourself who you would like to have on your team: someone who stops at learning the bare minimum to get paid and inevitably ruins the integrity of a product or someone who continually pursues getting better at their job and accelerates the people around them? Would you rather work with someone who values knowledge or someone who doesn’t?
I think it is drastically foolish to defend the toleration of those who aren’t even willing to learn basics like how a computer works in the name of inclusivity, utilitarianism, or egalitarianism.
Furthermore, I know a slew of extremely bright, tenacious, and committed young people who are ruthlessly pursuing excellence in engineering. If given the chance and enough time they will easily outperform, outwork, and surpass anyone who adheres to these lazy practices. If you do think this way and don’t want these kids to take your job then I suggest you buckle down a bit and learn some basics.
Look, invoking "straw man" is a bit of a cliche around here, but to say:
> As an example, how seriously would you take an aircraft flight controls software engineer who willfully chooses to never learn how an airplane flies? What if they just claimed that their sole responsibility was to take flight requirements from “people who know how airplanes work” and implement code?
is highly disingenuous given that learning to dig into what needs to be built is explicitly laid out in the article as the opposite of memorizing acronyms about how one might build it, and:
> I think it is drastically foolish to defend the toleration of those who aren’t even willing to learn basics like how a computer works in the name of inclusivity, utilitarianism, or egalitarianism.
seems to me to be far out of left field given that the article in question isn't speaking in the name of any of those things, but instead comparing to the value of other kinds of knowledge and expertise. If you feel the need to invoke "inclusivity" in this conversation, that only speaks to your own mindset.
I would much rather work with people who are able to analyze what kinds of knowledge actually drive their getting better results at their job than with people who assume going lower on the stack must equal "excellence in engineering".
> As an example, how seriously would you take an aircraft flight controls software engineer who willfully chooses to never learn how an airplane flies? What if they just claimed that their sole responsibility was to take flight requirements from “people who know how airplanes work” and implement code?
is highly disingenuous given that learning to dig into what needs to be built is explicitly laid out in the article as the opposite of memorizing acronyms about how one might build it, and:
> I think it is drastically foolish to defend the toleration of those who aren’t even willing to learn basics like how a computer works in the name of inclusivity, utilitarianism, or egalitarianism.
seems to me to be far out of left field given that the article in question isn't speaking in the name of any of those things, but instead comparing to the value of other kinds of knowledge and expertise. If you feel the need to invoke "inclusivity" in this conversation, that only speaks to your own mindset.
I would much rather work with people who are able to analyze what kinds of knowledge actually drive their getting better results at their job than with people who assume going lower on the stack must equal "excellence in engineering".
> As an example, how seriously would you take an aircraft flight controls software engineer who willfully chooses to never learn how an airplane flies?
This describes exactly the Boeing and MCAS fiasco. In fact, not only did Boeing hire do this, they outsourced the work overseas. People overseas are capable of doing great work, but they do the work exactly the the spec and nothing more. If you don't have an internal engineering process to account for situations unique to software, there's no way you can ensure that what they deliver is fit for purpose.
> Furthermore, I know a slew of extremely bright, tenacious, and committed young people who are ruthlessly pursuing excellence in engineering.
Those types of kids are going extinct. The kids coming up now are entirely smart-phone raised. An actual computer is an afterthought, they can't even explain what a file is. We're headed for the software engineering dark ages.
This describes exactly the Boeing and MCAS fiasco. In fact, not only did Boeing hire do this, they outsourced the work overseas. People overseas are capable of doing great work, but they do the work exactly the the spec and nothing more. If you don't have an internal engineering process to account for situations unique to software, there's no way you can ensure that what they deliver is fit for purpose.
> Furthermore, I know a slew of extremely bright, tenacious, and committed young people who are ruthlessly pursuing excellence in engineering.
Those types of kids are going extinct. The kids coming up now are entirely smart-phone raised. An actual computer is an afterthought, they can't even explain what a file is. We're headed for the software engineering dark ages.
I agree, as long as the attempt at expertise is not at the expense of getting stuff done. I have met people who have a great grasp of foundational principles and it shows. I have met other people who take so long trying to understand something before they use it, they don't get their work done.
You are so right about the pursuit of excellence being important but the industry is still way too fragmented. If a Doctor said they don't really need to know stuff, they would work it out as they went along, they wouldn't work for long. In Software Engineering, we are still treated (and act!) like some basic trade where the risks are low enough not to care.
You are so right about the pursuit of excellence being important but the industry is still way too fragmented. If a Doctor said they don't really need to know stuff, they would work it out as they went along, they wouldn't work for long. In Software Engineering, we are still treated (and act!) like some basic trade where the risks are low enough not to care.
How is an aircraft flight controls software a good example? I think everyone would agree someone building such software should know how a plane works. But do they have to be an aeronautical engineer? No. Where do you draw the line? There is a difference between flight control where lives are at stake and say spitting out a CRUD feature as fast as possible to see if anything sticks to reach market before the competition, then switching gears accordingly. If I deep-dived and skilled up to excellence on every technology I touched I'd still be on the first one. If all software were designed to NASA standards we would not be where we are now.
I don't expect a web designer to know how NAND gates or how to program in C because it's irrelevant. If they spent time learning that on the job, that's likely a waste of time. Abstractions exist for a reason.
I agree completely though that lazy people are out there. Professionals should always be learning and trying to improve their craft. Learning the bare minimum shouldn't be the default.
I don't expect a web designer to know how NAND gates or how to program in C because it's irrelevant. If they spent time learning that on the job, that's likely a waste of time. Abstractions exist for a reason.
I agree completely though that lazy people are out there. Professionals should always be learning and trying to improve their craft. Learning the bare minimum shouldn't be the default.
> When something goes horribly wrong, your mission critical system crashes, when you get dragged before the higher ups to explain the situation and fix it, the excuse ‘I don’t have to know how this works to do my job’ simply won’t fly.
At that point you just get another job and in the interview talk about how the company had bad practices that caused X bad thing.
At that point you just get another job and in the interview talk about how the company had bad practices that caused X bad thing.
Oh for fucks sake, every month this topic comes back. It honestly doesn't matter if the fundamentals are useful or not. What gets me is that programmers want to be hired by top-tier companies and earn massive salaries without putting a single drop of effort into actually acquiring the knowledge that justifies that position.
Can we just collectively swallow our privilege and stop whining that an employer is asking for proof that you've actually put the effort to be an engineer before they dump a river of money onto you? Look at how lucky you're before whining that the world is unfair, and shut the hell up about fundamentals.
The interviews surely aren't perfect, but the constant whining is ridiculous. Why not just study it and be done with it already?
Can we just collectively swallow our privilege and stop whining that an employer is asking for proof that you've actually put the effort to be an engineer before they dump a river of money onto you? Look at how lucky you're before whining that the world is unfair, and shut the hell up about fundamentals.
The interviews surely aren't perfect, but the constant whining is ridiculous. Why not just study it and be done with it already?
>an employer is asking for proof that you've actually put the effort to be an engineer before they dump a river of money onto you?
But that's the rub, knowing the acronym to ACID isn't proof of anything other than you read about it recently. Asking about the ACID test is like asking what Chipper Jones' batting average was in 1994 when interviewing to be a baseball player in 2021.
20+ years ago, I could recite all that shit from memory, but after building countless systems from the ground up which are still in use today, I can only remember reading about it but I can't recite the words of the acronym.
Hiring in IT is fundamentally broken and has been for decades and probably will be for decades. This is an example of it, that is the discussion.
But that's the rub, knowing the acronym to ACID isn't proof of anything other than you read about it recently. Asking about the ACID test is like asking what Chipper Jones' batting average was in 1994 when interviewing to be a baseball player in 2021.
20+ years ago, I could recite all that shit from memory, but after building countless systems from the ground up which are still in use today, I can only remember reading about it but I can't recite the words of the acronym.
Hiring in IT is fundamentally broken and has been for decades and probably will be for decades. This is an example of it, that is the discussion.
Sure, but my counter-point is: "who cares?" Why is it important for the process to the perfectly fair when there's a truck-load of money at play? As soon as you get hired you're set for life, so what's so terrible about having to spend some time learning about a couple of additional concepts? Even if you never use them again, you're still on the win by an unfathomable margin.
Not to be disrespectful to anyone, but this "issue" is the personification of that gif with a man wiping his tears with 100 dollar bills. The tech community has become increasingly disconnected from the real world in the past 10 years.
Not to be disrespectful to anyone, but this "issue" is the personification of that gif with a man wiping his tears with 100 dollar bills. The tech community has become increasingly disconnected from the real world in the past 10 years.
>it important for the process to the perfectly fair
It's not about fairness, it's about the ability for a company to hire what they need, and the person who best fits that need to be hired. That isn't happening now because the hiring process is broken. I think the main issue is people are afraid to fire employees who don't work out, although there are plenty of mechanisms (legal and policy) to make this frictionless. There is also a tsunami of unqualified people trying to get hired because it's one of the last few occupations that pays fairly well.
>As soon as you get hired you're set for life
That isn't anywhere close to true.
>The tech community has become increasingly disconnected from the real world in the past 10 years.
Sounds like you have an axe to grind. What are you basing this off of? I assume you don't work in tech.
It's not about fairness, it's about the ability for a company to hire what they need, and the person who best fits that need to be hired. That isn't happening now because the hiring process is broken. I think the main issue is people are afraid to fire employees who don't work out, although there are plenty of mechanisms (legal and policy) to make this frictionless. There is also a tsunami of unqualified people trying to get hired because it's one of the last few occupations that pays fairly well.
>As soon as you get hired you're set for life
That isn't anywhere close to true.
>The tech community has become increasingly disconnected from the real world in the past 10 years.
Sounds like you have an axe to grind. What are you basing this off of? I assume you don't work in tech.
[deleted]
This isn't about studying and going through it once - you have to study it, go through it with most companies you interview with, and then re-study it once you've not done it in your job for a while.
It isn't about putting in the effort. It's that even after you've provably put in the effort, they still don't believe that you've put in the effort, and so they constantly test you to make sure that you've put in the effort.
It isn't about putting in the effort. It's that even after you've provably put in the effort, they still don't believe that you've put in the effort, and so they constantly test you to make sure that you've put in the effort.
Because you aren't done with it. You need to re-learn it every year to get a raise. So about 20% of my time in this field is spent doing stuff exclusively for interviewing purposes.
The Feynman principle applies - it is important to know the thing and not as important to know it's name. In an interview, find out if the person knows them thing or can work it out.
> it is important to know the thing and not as important to know it's name
Yes and no. For years I did little computer programming projects on my own. Built some neat things, for my purposes at least. But because I didn't have the words to describe things I wasn't able to build a concrete image in my head. When I finally took a CS course and had the words to describe concepts I'd used for so long it was like a switch had been flipped, allowing me to think about problems in a much better way. I knew what many things were but because I didn't have the concepts mapped to concrete terms they remained fuzzy.
Yes and no. For years I did little computer programming projects on my own. Built some neat things, for my purposes at least. But because I didn't have the words to describe things I wasn't able to build a concrete image in my head. When I finally took a CS course and had the words to describe concepts I'd used for so long it was like a switch had been flipped, allowing me to think about problems in a much better way. I knew what many things were but because I didn't have the concepts mapped to concrete terms they remained fuzzy.
Though he does admit [1]
> So I had learned already that names don't constitute knowledge... That's caused me a certain trouble since because I refuse to learn the name of anything... What [my father] forgot to tell me was that the knowing the names of things is useful if you wanna talk to somebody else.
I think an interview might count :)
[1] https://www.youtube.com/watch?v=ga_7j72CVlc
> So I had learned already that names don't constitute knowledge... That's caused me a certain trouble since because I refuse to learn the name of anything... What [my father] forgot to tell me was that the knowing the names of things is useful if you wanna talk to somebody else.
I think an interview might count :)
[1] https://www.youtube.com/watch?v=ga_7j72CVlc
It's been sort of 18 years since I started using another heuristic - if you can google for something, don't bother requiring a candidate to know it. Not an exact principle, but somewhat like that .. in that I'm more curious about how a person thinks in an interview than what he/she knows.
This change in me was triggered when a candidate came in for a dev interview and we were doing the usual whiteboard problem thing back then. Someone on the panel asked "what would you do if you faced this problem on the job" and he said "I'd google for the solution". I recall not being able to make up my mind about whether to hire him right there or send him away :) .. but that was something I hadn't realized back then so the issue was not with him but with me. The lesson stayed.
So .. on the topic, if the candidate didn't know ACID .. an opportunity to google followed by a detailed discussion on it would work for me.
This change in me was triggered when a candidate came in for a dev interview and we were doing the usual whiteboard problem thing back then. Someone on the panel asked "what would you do if you faced this problem on the job" and he said "I'd google for the solution". I recall not being able to make up my mind about whether to hire him right there or send him away :) .. but that was something I hadn't realized back then so the issue was not with him but with me. The lesson stayed.
So .. on the topic, if the candidate didn't know ACID .. an opportunity to google followed by a detailed discussion on it would work for me.
It actually might be more informative. If a candidate, that had never heard the term "ACID", was able to Google it and talk intelligently about it I would be most impressed and it would probably earn them a next step. In fact, that might be a good practice to try out. Find a related term that the candidate wasn't familiar with and have them research it. Give them X minutes to reason about the term and then have a discussion about it. This is a thing we all do, every day. You can't possibly know everything so you must know how to find out about new things and how to think about them.
Yes indeed. The names are relevant for communication (i.e. to become a part of a "discourse community" [1]), but are not indicative of one's competence at a topic. They kind of go hand in hand often due to arranged education, but that's not essential.
Names also come handy when you need to search for something and search engines are not powerful at taking your description of what you're searching for.
[1] https://en.wikipedia.org/wiki/Discourse_community
Names also come handy when you need to search for something and search engines are not powerful at taking your description of what you're searching for.
[1] https://en.wikipedia.org/wiki/Discourse_community
Exactly. If someone does know what ACID compliance is that's great. We share a common language and that makes things easier. But if they don't, no big deal, but they should still know the concepts. There are people out there who don't know the difference between mongo and postgres. That's lacking the fundamentals.
Most of your time isn't spent in fundamentals though. I find a lot of my time is spent looking up some kind of best practice, or discovering what libs other people are using for a problem.
Also, things may be fundamental, but not everything is built up from fundamentals. Maybe math is the best place to look at this. When you go to school, you just do it. Number lines, equations, calculus, matrices. Even through the engineering course I did, which had all sorts of advanced topics, nobody mentioned ZFC or Peano or Cantor or anything like that. And while it's true that if you dig under the stuff that you get taught there's a whole interesting world, it doesn't seem to stop anyone from using math.
With coding, a lot of it is having the right attitude. Something's broken, I'm going to try to understand why and what other people have done to fix similar issues. There's a resourcefulness that beats classroom teaching in terms of usefulness. You want that person who is capable of understanding everything that might happen, what he needs to do to fix it, etc. That person is never going to know or have heard of everything to start with, but they can deal with things when they happen.
But of course it's easier to test whether someone has heard of this or that thing. Perhaps this is why senior devs find it easier to get a job than junior ones, employers are more likely to think that they can work through issues.
Also, things may be fundamental, but not everything is built up from fundamentals. Maybe math is the best place to look at this. When you go to school, you just do it. Number lines, equations, calculus, matrices. Even through the engineering course I did, which had all sorts of advanced topics, nobody mentioned ZFC or Peano or Cantor or anything like that. And while it's true that if you dig under the stuff that you get taught there's a whole interesting world, it doesn't seem to stop anyone from using math.
With coding, a lot of it is having the right attitude. Something's broken, I'm going to try to understand why and what other people have done to fix similar issues. There's a resourcefulness that beats classroom teaching in terms of usefulness. You want that person who is capable of understanding everything that might happen, what he needs to do to fix it, etc. That person is never going to know or have heard of everything to start with, but they can deal with things when they happen.
But of course it's easier to test whether someone has heard of this or that thing. Perhaps this is why senior devs find it easier to get a job than junior ones, employers are more likely to think that they can work through issues.
When you look up a best practice you often get conflicting opinions. Solid fundamentals may help you evaluate among them at times.
Very true. Also at times when you're looking stuff up you get told what the fundamentals are, pretty handy. Eg you are looking at data structures and the docs keep telling you what the Big-O is. So eventually you decide to look it up.
I will still never understand why I was asked to code an implementation of a binary tree for an interview once, something that is usually left to those who implement standard libraries or similar low-level needs, for a job that was basically writing user interfaces.
I asked this once as a take home for an interview. First, I copied the first hit for “implement binary tree in C” from google. I made sure it compiled and then deleted the implementation of the insert method. That was the take home. I wanted to know that the applicant could reimplement this one missing function and compile the project and not get immediately stuck.
Because an implementation of a binary tree is dead simple.
I have a computer science degree from 10 years ago and I couldn't tell you how to implement one. I vaguely know the general principles but I'd have to sit down and start from scratch on how to implement one.
Why? The only time I knowingly used them is with database indices. For everything else I do, they are an unnecessary implementation detail. It doesn't matter what Ruby's Set uses.
Why? The only time I knowingly used them is with database indices. For everything else I do, they are an unnecessary implementation detail. It doesn't matter what Ruby's Set uses.
People who make this comment, I generally don't believe you, or don't believe you have done any programming. If I put a gun to your head and give you an hour, you can't make a node that points to two other nodes of the same type?
If you really can't, you should get familiar with how data can point to other data, it is fundamental and done all the time.
If you really can't, you should get familiar with how data can point to other data, it is fundamental and done all the time.
> If I put a gun to your head and give you an hour, you can't make a node that points to two other nodes of the same type?
It's easy to make an imbalanced binary tree.
I once decided that building a binary tree based only on a friend's explanation of how they worked; it seemed like it would be a good learning exercise for myself (I don't have a CS degree). It was like 10 minutes of coding to get it minimally working & tested, but then I had to work out all of the different rotations to make it a balanced binary tree (I don't know that an imbalanced binary tree is ever actually useful, as it could just work out to be a slower linked-list in the worst case scenario).
I did it once, took me a little while, but I don't know that I could do it again without having to reexamine all of the different cases that result in imbalance and the rotations that respond to each.
It's easy to make an imbalanced binary tree.
I once decided that building a binary tree based only on a friend's explanation of how they worked; it seemed like it would be a good learning exercise for myself (I don't have a CS degree). It was like 10 minutes of coding to get it minimally working & tested, but then I had to work out all of the different rotations to make it a balanced binary tree (I don't know that an imbalanced binary tree is ever actually useful, as it could just work out to be a slower linked-list in the worst case scenario).
I did it once, took me a little while, but I don't know that I could do it again without having to reexamine all of the different cases that result in imbalance and the rotations that respond to each.
Binary search tree is more difficult than a binary tree. A BST is a BT but a BT is not a BST. So making a normal BT is just pointing nodes.
Huh... TIL, as they say.
There’s more than making a node the points to other nodes. That’s not what makes a binary tree a binary tree.
I mean… it literally is all there is to it
Well, the requirements I faced were to implement the tree such that nodes were inserted with the standard ordering you would expect. Walking the tree to find the best place to insert them, etc.
Again, that's stupidly easy. Right is bigger, left is smaller.
If the current node is a leaf, insert left if smaller and right if bigger.
If not a leaf nods, run the method recursively on the left it it's smaller, on the right if it's bigger.
If they asked you to write a self-balancing binary tree it's one thing, but there's nothing tricky about writing a normal ordered binary tree.
If the current node is a leaf, insert left if smaller and right if bigger.
If not a leaf nods, run the method recursively on the left it it's smaller, on the right if it's bigger.
If they asked you to write a self-balancing binary tree it's one thing, but there's nothing tricky about writing a normal ordered binary tree.
It isnt the knowledge itself, it is making sure it works and is tested, etc, all in <45m. It serves as a worse fizzbuzz imo.
Do you want me to show you a single-linked list or a double-linked one?
[data, *next] [data, *next] [data, *next] [data, *next]
This is an arrogant comment. You are right that the general algorithm of a binary tree is not complicated. But coding it under pressure, for anyone who is not writing that kind of code on a daily basis, which is most people, and especially for a job where you’re not required to write that kind of code, is ridiculous.
If you audition for the New York City ballet, they don’t expect you to ice skate.
If you audition for the New York City ballet, they don’t expect you to ice skate.
Not to those who struggle with the notion of a pointer (or other such “fundamentals”), it’s not.
There are loads of simple tasks, there should be a bit more reasonaing as to why this is a good filter relevant to the job's day to day.
For example I think writing some logic to filter datapoints or timespans is much better and more practical. But it takes more effort than just opening and pocking binary tree.
For example I think writing some logic to filter datapoints or timespans is much better and more practical. But it takes more effort than just opening and pocking binary tree.
The fact that you didn't even think about asking if it's balanced or not, and just went into declaring stupid everyone that considers the problem hard is annoying.
I wouldn't want to have an interview with you, or you selecting people for a team I'm on.
I wouldn't want to have an interview with you, or you selecting people for a team I'm on.
Why would it be balanced. Nobody said it was a search tree.
Just the fact that somebody is talking about "implementing" it implies there's further complication than defining a recursive type.
People in general do not communicate on the cleanest ambiguosity-free way;
People in general do not communicate on the cleanest ambiguosity-free way;
If you can’t remember the details or understand what makes the problem hard I doubt it would’ve been fine even given a whole day.
I’ve interviewed a _lot_ of people. The people who gripe the loudest about algorithms haven’t had a chance just from answers to situational questions.
I’ve interviewed a _lot_ of people. The people who gripe the loudest about algorithms haven’t had a chance just from answers to situational questions.
You mean a naive one, that is, a useless one...or a practical one? Such as, at least something like LLRB?
A binary tree is just a tree data structure where each node has at most two children. And that's dead simple and every programmer should be able to implement that when aksed. Note that it doesn't even have the search (i.e. sorted) requirement and much less the self-balancing one. You can work your way towards that with the right questions and that would be an acceptable way to structure an interview. Start with something simple everyone should be able to do and then work your way up the requirements and discuss potential problems/pitfalls and their solutions.
Nobody talked about implementing a specific variant of a red-black tree when the general topic of self-balancing binary search tree hasn't even be mentioned.
Nobody talked about implementing a specific variant of a red-black tree when the general topic of self-balancing binary search tree hasn't even be mentioned.
To add to that, a binary tree is often not really "implemented" but simply encoded in an array. That is, the children of any given node i are 2i and 2i+1 in the array.
So depending on the actual exercise/interview question you might able to just show that, especially of you have to use pseudo code.
So depending on the actual exercise/interview question you might able to just show that, especially of you have to use pseudo code.
Correct me if I'm wrong, but isn't this exactly the case where you don't want an unbalanced tree? A degenerate tree is bad enough; a degenerate tree implemented as an array in this way is catastrophic. So it seems to me that if you're refining the topic further in some sort of talk, the topic of balancing should precede the topic of physically representing the tree as an array.
It all depends on the specific topic. If I just want to store my data in binary tree structure without the possibility of deletes and just with an insertion-order requirement, then the tree is automatically balanced and using an array like that is totally fine. The use-cases are probably rare (I can't think of a situation from the top of my head), but binary tree doesn't have to mean search tree.
The self-balancing part is usually added after the search part as balance is trivial/unimportant when the order/position of the elements has no constraints.
The self-balancing part is usually added after the search part as balance is trivial/unimportant when the order/position of the elements has no constraints.
> The use-cases are probably rare (I can't think of a situation from the top of my head)
I imagine there's quite a lot of them. Things like VP-trees and such get often built statically and could be implemented in a similar way. Not sure if the simple (2i, 2i+1) schema is optimal for this but perhaps it could be adapted.
I imagine there's quite a lot of them. Things like VP-trees and such get often built statically and could be implemented in a similar way. Not sure if the simple (2i, 2i+1) schema is optimal for this but perhaps it could be adapted.
Yes, you are of course correct. I was thinking of a heap, which is a complete binary tree.
It is dead simple. What you're describing (two children, no constraints) is basically a cons cell. Yet I suspect that most people don't use cons cells as binary trees. Which is probably why I didn't understand that apparently rather nebulous topic as being this generic.
This is another point in which this can be a good interviewing question. For certain jobs you don't want someone who just forges ahead with an unclear specification.
That said, for most dev jobs implementing a binary tree isn't necessary knowledge. But if I need a dev who can get their hands dirty I would be weary of hiring someone who can't take a decent stab at it during an interview.
That said, for most dev jobs implementing a binary tree isn't necessary knowledge. But if I need a dev who can get their hands dirty I would be weary of hiring someone who can't take a decent stab at it during an interview.
Because sometimes the requirement is that the engineer being hired is able to come up with efficient algorithms. You don't have to be able to create the textbook binary tree algorithms, but you should be able to derive a reasonably performant algorithm with simple data structures if it matters to the position you're being hired for.
> requirement is that the engineer being hired is able to come up with efficient algorithms.
Coding binary tree is not the way to discover that.
Presenting a task and observing people solving it and explaining computation complexity behind their solution and the thinking behind their solution is how you do that.
Coding binary tree is not the way to discover that.
Presenting a task and observing people solving it and explaining computation complexity behind their solution and the thinking behind their solution is how you do that.
It's a fine way of doing it, although certainly it shouldn't be the only algorithms question.
It's a simple data structures / algorithms question. Can you think in terms of how data is stored in memory? Do you understand the tradeoffs of storing data in one form or another? Can you work with recursive algorithms? Given a challenge of having to think about data in a form you haven't recently, can you wrap your mind around the characteristics of that data structure?
If you're getting asked all this for a ReactJS UI building job, then the interviewer is probably off-base. But otherwise, make sure you're applying for jobs you're suited for. If you don't want a job where coming up with efficient algorithms is important, that's fine - there's plenty of others out there.
It's a simple data structures / algorithms question. Can you think in terms of how data is stored in memory? Do you understand the tradeoffs of storing data in one form or another? Can you work with recursive algorithms? Given a challenge of having to think about data in a form you haven't recently, can you wrap your mind around the characteristics of that data structure?
If you're getting asked all this for a ReactJS UI building job, then the interviewer is probably off-base. But otherwise, make sure you're applying for jobs you're suited for. If you don't want a job where coming up with efficient algorithms is important, that's fine - there's plenty of others out there.
> Can you think in terms of how data is stored in memory?
If you're using a modern OS, you don't know this. The kernel remaps everything and pretends to give you a linear chunk of memory. You don't know what's in the various caches and what's been paged out of ram completely.
Everything is built on layers of abstraction. C is an abstraction layer that fits poorly into what modern CPUs and operating systems actually do, but everything is programmed to cater to C's assumptions so people don't need to rewrite their programs.
Higher languages don't necessarily use the data structure you think they do and they can switch the implementation details based on the size.
"Simple data structures" are only simple because you pick one specific layer of abstraction to work with.
If you're using a modern OS, you don't know this. The kernel remaps everything and pretends to give you a linear chunk of memory. You don't know what's in the various caches and what's been paged out of ram completely.
Everything is built on layers of abstraction. C is an abstraction layer that fits poorly into what modern CPUs and operating systems actually do, but everything is programmed to cater to C's assumptions so people don't need to rewrite their programs.
Higher languages don't necessarily use the data structure you think they do and they can switch the implementation details based on the size.
"Simple data structures" are only simple because you pick one specific layer of abstraction to work with.
That's not an excuse for not being able to reason about data structures if your job requires working with data structures and algorithms. The logical extension of your argument is "put everything in a linked list and let the OS work it out."
If I give a candidate a truly real world difficult problem with an hour or even a day to solve it, some people who are not so smart will get the correct answer by luck, some people who are brilliant will start off with a bad approach and not get there.
If I ask if you can do something basic binary tree things, I will also not know if you are brilliant, but I can at least rule out if you are totally clueless or not.
If I ask if you can do something basic binary tree things, I will also not know if you are brilliant, but I can at least rule out if you are totally clueless or not.
Sorry, but regurgitating binary tree operations is not "coming up with efficient algorithms".
Well hopefully there's some layering in the question if it matters to the job. If you have an engineering position in which a high degree of CS competence is required and the candidate can't even make it past the initial layer of "regurgitating binary tree operations" then you have the wrong candidate.
> If you have an engineering position in which a high degree of CS competence is required and the candidate can't even make it past the initial layer of "regurgitating binary tree operations" then you have the wrong candidate.
Or you have a candidate who understands binary trees, can implement it but can't regurgitate binary tree operations from the top of their memory.
Or you have a candidate who understands binary trees, can implement it but can't regurgitate binary tree operations from the top of their memory.
For me the point would be that a no-frills binary tree is simple enough that any dev qualified for such a position should be able to figure out what needs to be done from the specification, without needing to regurgitate anything.
If they can't recall how balancing is done that would be fine with me, as long as they mention it.
If they can't recall how balancing is done that would be fine with me, as long as they mention it.
Exactly. And that's the reason you'd probably layer the question with some more complex asks after the initial answer. If somebody has memorized the algorithm but then can't explain further, it's obvious they've "regurgitated" without really having a deep understanding of the problem being solved with the data structure. By contrast, a good candidate may not be able to immediately repeat the algorithm, but can reason their way there pretty quickly.
> By contrast, a good candidate may not be able to immediately repeat the algorithm, but can reason their way there pretty quickly.
I know dozens of good candidates who can't get there pretty quickly but are really good. I also know dozens of good candidates who can get there pretty quickly but won't be good enough on the job.
I know dozens of good candidates who can't get there pretty quickly but are really good. I also know dozens of good candidates who can get there pretty quickly but won't be good enough on the job.
Maybe they'd be really good at other engineer positions, but if they can't get to a binary tree algorithm reasonably fast I wouldn't hire them for backend positions that require CS reasoning.
> Maybe they'd be really good at other engineer positions, but if they can't get to a binary tree algorithm reasonably fast I wouldn't hire them for backend positions that require CS reasoning.
You not hiring for backend positions doesn't mean they are not good at the positions. It just means they aren't good at your signals.
You not hiring for backend positions doesn't mean they are not good at the positions. It just means they aren't good at your signals.
For me it's a strong signal, but it shouldn't be the only signal. And I do think it's good to have a diverse team.
Some of my coworkers are not nearly as strong as me when it comes to "CS details" like data structures or programming languages, but they have other attributes which complement me and which are good for the team.
Having a bunch of me's (or any of the others) would be a disaster.
Some of my coworkers are not nearly as strong as me when it comes to "CS details" like data structures or programming languages, but they have other attributes which complement me and which are good for the team.
Having a bunch of me's (or any of the others) would be a disaster.
You mean, "regurgitate" operations like this?
find = lambda k, t: t if t is None or k == t.k else find(k, t.left) if k < t.k else find(k, t.right)
That was probably a question to determine basic competency in coding, simple data structures and simple algorithms. I.e. if he can do this, that means he's at least minimally competent. if he can't, then we don't have to waste any more time.
Because it is the 2nd easiest possible data structure to implement, if you can't do that it means you don't grasp how data can point to other data and how you can leverage recursion to make hard problems easy to solve. Perhaps they were hiring for someone that they needed to be capable of more than just trivial CRUD wiring. Even when you aren't writing core libraries these things come up constantly, sometimes at higher levels when designing systems and how they interact even.
I implore you get to a state where this isn't hard for you, it will make you capable of more.
I implore you get to a state where this isn't hard for you, it will make you capable of more.
I mean sure, yes, there's also programmers out there who've been working for years and look at me like I'm speaking in tongues when I talk about closure.
The author's casual arrogance about "pixel pushers" only caring about "hotdog stand colors" is just as damaging to the profession -- as it infects the hardest part of getting things right most of the time: teamwork.
The author's casual arrogance about "pixel pushers" only caring about "hotdog stand colors" is just as damaging to the profession -- as it infects the hardest part of getting things right most of the time: teamwork.
I think you misunderstood his point, I think what he's saying is that even good back end devs are horrible at designing UI and you need different people with different specialisations on a team.
I'm horrible at designing UI too. That's simply not the job of building modern interfaces across multi-disciplinary teams.
I care far, far more about data-structures, eventual-consistency, data-design, and types than CSS and divs.
These days getting the design to look nice is my two-hour reward for nine days of data-stitching hell, writing backend-for-frontend lambda functions, and negotiating API contracts that actually support the product I've been asked to build.
We need to stop assuming that just because a programmer works on client code, that they are in any way less capable that a programmer that works on server code. My point stands: It's absolutely toxic to collaboration, which is the key to success.
I care far, far more about data-structures, eventual-consistency, data-design, and types than CSS and divs.
These days getting the design to look nice is my two-hour reward for nine days of data-stitching hell, writing backend-for-frontend lambda functions, and negotiating API contracts that actually support the product I've been asked to build.
We need to stop assuming that just because a programmer works on client code, that they are in any way less capable that a programmer that works on server code. My point stands: It's absolutely toxic to collaboration, which is the key to success.
I think he was saying that the GUI programmer doesn't need to know how to use a profiler for their work to be valuable, and that if you're good at optimizing you don't need to know how to choose tasteful colors for your work to be valuable. My reading is the opposite of your reading.
Yes the author was criticizing the kind of "I have deep CS knowledge, you are just pushing pixels" arrogance.
The part which was more problematic for me was the correlation between having a comp-sci degree and working backend, it felt like a subtle form of gate keeping i.e. you could interpret this as: haven't got a degree? you will be working frontend/ui.
[deleted]
“Atomicity, something, something, Durability?”
My exact response after reading the first quote. It is indeed fundamental, and something I am mindful of with every database query I execute. But I haven’t thought about the exact definition and its mnemonic in over a decade.
My exact response after reading the first quote. It is indeed fundamental, and something I am mindful of with every database query I execute. But I haven’t thought about the exact definition and its mnemonic in over a decade.
I don't think anyone reasonable actually suggest that one has to memorize the exact book definitions, but simply to be demonstrate that they understand the important parts.
Not being able to give any explanation or sign that you've at least heard of it is a completely different story from not being able to recall what the letter C stands for (consistency IIRC?).
As the saying goes: If you think you understand something, and you can't explain it to someone else, that you probably don't really understand it.
Not being able to give any explanation or sign that you've at least heard of it is a completely different story from not being able to recall what the letter C stands for (consistency IIRC?).
As the saying goes: If you think you understand something, and you can't explain it to someone else, that you probably don't really understand it.
If I was hiring your response would be sufficient. Assuming you could expound a bit on the point of it and offer some example scenarios. So many interview objections are about extreme interpretations of these questions, which do happen but are not normal.
Yeah, I assume the quoted interviewer attempted to further clarify the question and its intent. “Explain ACID” might be misheard or misunderstood by the subject. “Well, I don’t know, I guess I get reflux any time I’m doing an interview.“
> There is another fundamental that is ignored by a lot of computer users. I think everyone who spends a lot of time
behind a computer should learn how to touch type, since typing is still the main way of interacting with your computer.
I sort of agree, but anyone typing out code at 120WPM isn't programming, they're writing. Programming is an intellectual activity, not a physical one so learning to touch type is worthwhile but might only gain you a few minutes a day of productivity over someone doing 40WPM in a home-grown hunt-and-peck style of typing?
I sort of agree, but anyone typing out code at 120WPM isn't programming, they're writing. Programming is an intellectual activity, not a physical one so learning to touch type is worthwhile but might only gain you a few minutes a day of productivity over someone doing 40WPM in a home-grown hunt-and-peck style of typing?
You touch type not to break a world record. You touch type because you can write without thinking about the keyboard, just like you don't think when using or arms or legs, and it's faster to correct an error that you made.
Imagine that you want to correct a block of code or a whole paragraph. If you don't touch type, maybe you will think twice before rewriting it.
Imagine that you want to correct a block of code or a whole paragraph. If you don't touch type, maybe you will think twice before rewriting it.
I don't mind being slow early on when I'm doing something new, because I know that I will get muscle memory for things I repeat a lot and become faster. I also use an editor that supports mapping keys (currently vim).
Besides, only Java can make any programming task I/O bound ;p
Besides, only Java can make any programming task I/O bound ;p
Programming is a uniquely bad fit for formal touch typing because we type a lot of symbols that don't fit on the home rows. I'm typing this comment without looking at the keyboard, but I'm not "touch typing" because my hands aren't kept on the home row.
Need to use the arrow keys? Outside the touch typing range. Mouse? Outside touch typing. Function keys? Numpad? Touchbar (blech)?
I have never felt limited by typing speed. Informal touch typing is good enough. Quite often it's the computer or network that can't keep up, anyway.
Need to use the arrow keys? Outside the touch typing range. Mouse? Outside touch typing. Function keys? Numpad? Touchbar (blech)?
I have never felt limited by typing speed. Informal touch typing is good enough. Quite often it's the computer or network that can't keep up, anyway.
I've seen this argument before and I don't think I agree. Surely there's a lot of times when I program that are heavy into thinking, but there's also a lot of mechanical work. Refactoring? Typing speed will help you. Going to try out a new approach? Typing speed will help you to see whether it's true or not.
Everything else being equal, typing speed will help you try out stuff faster.
Everything else being equal, typing speed will help you try out stuff faster.
I'm a developer who is also a pretty fast typist, and I have to say that's not my experience. I wouldn't want to hunt and peck, but my sense is that higher than, like, 40 GWAM probably yields diminishing returns.
Language might make a difference there. Writing C# with a QWERTY keyboard, you still have to interrupt a normal typing flow for your parentheses, braces, etc.
I suppose it's helpful if you're writing long comments, but for the most part, I rarely hit sustained bursts of my peak typing speed when writing code.
Language might make a difference there. Writing C# with a QWERTY keyboard, you still have to interrupt a normal typing flow for your parentheses, braces, etc.
I suppose it's helpful if you're writing long comments, but for the most part, I rarely hit sustained bursts of my peak typing speed when writing code.
Also typing fast does not mean writing fast. Think about the speed tests you took. You probably kind of zoned out and just bashed it out. But when actually writing/coding you stop and think about it a bit. I used to hit 70-80. But now I maybe manage around 20-40, probably less. Because I have to think at the same time and can not just 'zone out' and copy. Knowing the keys is a massive help though. With coding it is more 'i know an interface that does something'. Ok start using this object oh yeah wait it is a pointer not a local so different call syntax, oh and then the call I need is inside that object but that is local to that object, oh wait lets go read the docs this thing has 4 params and that 3rd one is always a bit tricky... No way you are hitting 40+wpm unless you just happen to use only calls you use all the time and have memorized all of the params that go into them and have the code basically already written in your head.
I'd go a step further. In my experience writing C# I don't really need to type. Like I'll just sit on CTRL + Space and the arrow keys most of the time.
To be controversial, a language that rewards typing speed seems to me like a failure of tooling, why am I manipulating text like a caveman (ok.. ok.. Mesopotamian) rather than with the symbols that the language actually deals with.
To be controversial, a language that rewards typing speed seems to me like a failure of tooling, why am I manipulating text like a caveman (ok.. ok.. Mesopotamian) rather than with the symbols that the language actually deals with.
I put my parentheses on the [] keys: http://canonical.org/~kragen/setting-up-keyboard
It helps a lot with most programming languages. Snippets for if, while, and for can also help with the {}.
It helps a lot with most programming languages. Snippets for if, while, and for can also help with the {}.
Way too many times I have seen fast typers who would spend as much time adding text as immediately deleting it after that and typing something else in its place, over and over. (Maybe that’s the way they think, I don’t know.)
It is for me - sometimes trying out/typing out a solution is the fastest way for me to get a feel for if it'd actually work in practice.
All else being equal sure typing faster is only going to make you more productive but a slower typer who is better at solving complex problems is always going to be more valuable than a fast typer who is not as good. I don’t think I’ve ever encountered a programmer that was too slow to simply be productive and valuable to a business which makes it questionable advice for anyone to follow.
It's about touch typing only. If you start changing other variables, the comparison doesn't make sense anymore (to the discussion).
Of course your senior engineer that wrote the thing will be more productive than your intern that writes 1000 WPM.
Of course your senior engineer that wrote the thing will be more productive than your intern that writes 1000 WPM.
This is a false dichotomy. Being a great problem-solver doesn't disqualify one from being a fast typist.
I think that hunting for the correct key is distracting.
I am pretty sure that people who didn't learn to type properly have to switch their whole mind frame from the content into the typing, back and forth.
Some people even speak outloud, or at least move their lips according to, whatever they are typing.
I am pretty sure that people who didn't learn to type properly have to switch their whole mind frame from the content into the typing, back and forth.
Some people even speak outloud, or at least move their lips according to, whatever they are typing.
Yeah... even if you don't have to type fast, not needing to look at the keyboard when you type will at least help you to concentrate on what's on your screen and stay in the "flow".
Optimizing an outer loop also will help.
Does documentation factor into this?
If I can touch-type at 120WPM, then I can churn out a great deal more documentation per unit time than another engineer that can only type at 40WPM. It's less burdensome for me to document well, to write expositional documents and to communicate regularly than my colleague that needs to budget their typing more carefully.
While I might not be bottlenecked when working on a tricky piece of code at 40WPM, I'll likely be bottlenecked elsewhere often.
If I can touch-type at 120WPM, then I can churn out a great deal more documentation per unit time than another engineer that can only type at 40WPM. It's less burdensome for me to document well, to write expositional documents and to communicate regularly than my colleague that needs to budget their typing more carefully.
While I might not be bottlenecked when working on a tricky piece of code at 40WPM, I'll likely be bottlenecked elsewhere often.
Typing fast is useful but not necessarily enhancing productivity all that much. Even writing documentation is more about thinking than typing fast. Documentation must be concise and understandable.
I don’t know, when the thought about how to structure something is in my head, I want to type it out as fast as possible. My brain isn’t ideally suited to keeping complex constructs at the top of my thoughts for a long time.
There's also an argument here about having (and signalling) respect for one's peers, craft and tools.
The keyboard is a tool.
To me, it feels disrespectful for a person to expect these high wages when they don't even make the moderate effort involved in learning to use this tool efficiently.
In my experience, this usually signals that a person isn't actually interested in continuously improving, feels overly entitled and they tend to be a nightmare to work with.
Most places I've worked require substantial pair programming or at least working through problems together on a regular basis.
If you can't type, you're going to waste a lot of people's time and they may be too polite to say anything, but it's frustrating for everyone around you.
A carpenter who claims to be experienced, but can't work a hammer efficiently would be ridiculed on a job site.
I'm not proposing we ridicule anyone, but being unwilling to type and claiming to be competent at a job that requires an enormous amount of typing does have a ridiculous look (to me).
Of course, I have worked with people working through a variety of disabilities, including dyslexia that make touch typing basically impossible. I'm not talking about them here.
I'm talking about people who just refuse to make the effort for whatever reason. It's a bad look.
PSA for newer players: Regardless of how you may feel about this subject, failing to demonstrate at least moderate typing skill in an interview is often a hard pass unless there's some explanation requiring an ADA accomodation.
The keyboard is a tool.
To me, it feels disrespectful for a person to expect these high wages when they don't even make the moderate effort involved in learning to use this tool efficiently.
In my experience, this usually signals that a person isn't actually interested in continuously improving, feels overly entitled and they tend to be a nightmare to work with.
Most places I've worked require substantial pair programming or at least working through problems together on a regular basis.
If you can't type, you're going to waste a lot of people's time and they may be too polite to say anything, but it's frustrating for everyone around you.
A carpenter who claims to be experienced, but can't work a hammer efficiently would be ridiculed on a job site.
I'm not proposing we ridicule anyone, but being unwilling to type and claiming to be competent at a job that requires an enormous amount of typing does have a ridiculous look (to me).
Of course, I have worked with people working through a variety of disabilities, including dyslexia that make touch typing basically impossible. I'm not talking about them here.
I'm talking about people who just refuse to make the effort for whatever reason. It's a bad look.
PSA for newer players: Regardless of how you may feel about this subject, failing to demonstrate at least moderate typing skill in an interview is often a hard pass unless there's some explanation requiring an ADA accomodation.
> The keyboard is a tool.
Yes, but one of many.
> If you can't type, you're going to waste a lot of people's time and they may be too polite to say anything, but it's frustrating for everyone around you.
I don't think in my career I have ever been alerted to anyone's typing speed being slow.
Yes, I've heard endless noise about "the perfect keyboard" and the ins and outs of spring types and junk. But I've never noticed the output of someone because they type slowly.
And to be honest, I really couldn't care. Can they code? can they keep to a deadline? are they nice to work with?
if the answer to all three is yes, then I couldn't give a shit about how they type.
> but can't work a hammer efficiently would be ridiculed on a job site.
Only if they need to use a hammer effectively though. If they are laying veneer, carving, using a screwdriver like a sensible person, then a hammer is only rarely needed.
This all smacks of gatekeeping to me.
Yes, but one of many.
> If you can't type, you're going to waste a lot of people's time and they may be too polite to say anything, but it's frustrating for everyone around you.
I don't think in my career I have ever been alerted to anyone's typing speed being slow.
Yes, I've heard endless noise about "the perfect keyboard" and the ins and outs of spring types and junk. But I've never noticed the output of someone because they type slowly.
And to be honest, I really couldn't care. Can they code? can they keep to a deadline? are they nice to work with?
if the answer to all three is yes, then I couldn't give a shit about how they type.
> but can't work a hammer efficiently would be ridiculed on a job site.
Only if they need to use a hammer effectively though. If they are laying veneer, carving, using a screwdriver like a sensible person, then a hammer is only rarely needed.
This all smacks of gatekeeping to me.
Thanks for taking the time to respond.
I agree that some (maybe lots of?) people feel differently about this and of course, that's okay too.
To be clear, I'm not saying anyone needs to type 120 WPM, I'm talking about a single finger on each hand, eyes down, hunting for the next key to press.
Over the 20+ years of my career, I've been exposed to hundreds, maybe thousands of different folks doing different kinds of computer work.
And yes, I agree that there are likely people who can kind of get the job done without being able to operate a keyboard efficiently. I also agree, that it helps a lot if they're nice to work with.
But I have not yet met a single hacker who has demonstrable and deep knowledge of computers and the software that runs them who cannot type.
I'm sure they must exist, and it wouldn't surprise me if Donald Knuth himself dropped into this thread to let me know that he hunts and pecks!
I'm just saying, that I have not personally met or observed someone operating at the highest levels of this field who cannot type with a reasonable speed.
[update] Almost forgot about Gatekeeping, Yes!
Gatekeeping is an essential part of interviewing. In fact, that's the entire point. We interview people for roles with the specific intent of bringing on people who we believe (or at least hope) have higher competence and closing the gate to people who have lower competence.
It's incredibly difficult work and it's not perfect and we have a lot of work to do, to make it better and more inclusive, but expecting competence in basic, role-relevant skills is unambiguously reasonable IMO.
I agree that some (maybe lots of?) people feel differently about this and of course, that's okay too.
To be clear, I'm not saying anyone needs to type 120 WPM, I'm talking about a single finger on each hand, eyes down, hunting for the next key to press.
Over the 20+ years of my career, I've been exposed to hundreds, maybe thousands of different folks doing different kinds of computer work.
And yes, I agree that there are likely people who can kind of get the job done without being able to operate a keyboard efficiently. I also agree, that it helps a lot if they're nice to work with.
But I have not yet met a single hacker who has demonstrable and deep knowledge of computers and the software that runs them who cannot type.
I'm sure they must exist, and it wouldn't surprise me if Donald Knuth himself dropped into this thread to let me know that he hunts and pecks!
I'm just saying, that I have not personally met or observed someone operating at the highest levels of this field who cannot type with a reasonable speed.
[update] Almost forgot about Gatekeeping, Yes!
Gatekeeping is an essential part of interviewing. In fact, that's the entire point. We interview people for roles with the specific intent of bringing on people who we believe (or at least hope) have higher competence and closing the gate to people who have lower competence.
It's incredibly difficult work and it's not perfect and we have a lot of work to do, to make it better and more inclusive, but expecting competence in basic, role-relevant skills is unambiguously reasonable IMO.
> But I've never noticed the output of someone because they type slowly.
I do, when I'm pairing with someone, or in any synchronous meeting environment. It may not effect the 'output' of delivered code (for a developer) but it can hinder their ability to pair effectively.
I do, when I'm pairing with someone, or in any synchronous meeting environment. It may not effect the 'output' of delivered code (for a developer) but it can hinder their ability to pair effectively.
Typing should be a very small amount of time of a developer's job even when pair programming.
>A carpenter who claims to be experienced, but can't work a hammer efficiently would be ridiculed on a job site.
It's just not remotely the same. How quickly and accurately a carpenter can hammer has a large impact on the quality and speed of the work, while typing should never be the bottleneck for a developer
>A carpenter who claims to be experienced, but can't work a hammer efficiently would be ridiculed on a job site.
It's just not remotely the same. How quickly and accurately a carpenter can hammer has a large impact on the quality and speed of the work, while typing should never be the bottleneck for a developer
Would you say I am being disrespectful because when I learned to type at the age of 12 I wasn't following any official guide and learned to touch type with only ~6 fingers? I type at around 120 WPM on average. Is it just the speed, or do I have to conform to some specific definition of "touch typing?"
Of course the particular technique is irrelevant (to me at least).
I most enjoy working with people who care deeply about their work, and in my experience, those people are often (not always), people who also care about being efficient. In fact, if you're writing code at 120 wpm, maybe you've found a technique worth sharing and teaching! :-)
I most enjoy working with people who care deeply about their work, and in my experience, those people are often (not always), people who also care about being efficient. In fact, if you're writing code at 120 wpm, maybe you've found a technique worth sharing and teaching! :-)
Yep. You can’t find the best programmer in the room by measuring everyone’s typing speed. But the person with the slowest typing speed in the room will rarely be the best programmer. Unless they have RSI or something, typing slowly is just a simple, low effort, obvious signal someone hasn’t spent many hours in front of a computer yet. Programming skill takes time.
But how do you know if someone has dyslexia or is just not making an effort? Or has wrist, shoulder, whatever problems? Unless the typing is hurting their output, who cares? And at that point it is not the typing speed that is the problem, but that their output is low.
To be clear, I'm not insinuating that anyone needs to type 120WPM. I'm talking about single-finger, head down, hunting for the location of a key.
People with mild to moderate dyslexia (in my experience) and people with injuries are generally able to demonstrate competence with the keyboard and computer operation.
I have dealt with moderate to severe RSI, and I have worked with a relatively large number of people dealing with RSI over the years and in my experience injuries don't mask operational competence. These behaviors look different from, "where was that key again?"
Dyslexia often surfaces in a variety of early communications and over the years, working with some especially brilliant people challenged with dyslexia, I've learned to assume (early on) that typos are not a signal for carelessness, but potential indicators of dyslexia.
I'm aware dyslexia is a spectrum, and at the extreme it can be fully debilitating for some activities. I haven't had much exposure to that cohort, but I would be happy to make reasonable accommodations and certainly would not consider it disqualifying.
Thanks for bringing this up, I think it's helpful to be able to clarify these details!
People with mild to moderate dyslexia (in my experience) and people with injuries are generally able to demonstrate competence with the keyboard and computer operation.
I have dealt with moderate to severe RSI, and I have worked with a relatively large number of people dealing with RSI over the years and in my experience injuries don't mask operational competence. These behaviors look different from, "where was that key again?"
Dyslexia often surfaces in a variety of early communications and over the years, working with some especially brilliant people challenged with dyslexia, I've learned to assume (early on) that typos are not a signal for carelessness, but potential indicators of dyslexia.
I'm aware dyslexia is a spectrum, and at the extreme it can be fully debilitating for some activities. I haven't had much exposure to that cohort, but I would be happy to make reasonable accommodations and certainly would not consider it disqualifying.
Thanks for bringing this up, I think it's helpful to be able to clarify these details!
40WPM is fine, and far beyond actual hunt and peck.
But I have encountered one IT professional (not a programmer) who despite being in the industry for 20+ years, could only manage about 15WPM, if that. His style seemed to include legitimate hunting for the next key, and only two fingers.
At that speed it was starting to become problematic: It was painfully slow to watch him do anything or help him with a problem.
Not an absolute deal breaker, as long as such a person is good at their job and is efficient with what they type (taking advantage of copy/pasting or scripting), it's workable.
But I have encountered one IT professional (not a programmer) who despite being in the industry for 20+ years, could only manage about 15WPM, if that. His style seemed to include legitimate hunting for the next key, and only two fingers.
At that speed it was starting to become problematic: It was painfully slow to watch him do anything or help him with a problem.
Not an absolute deal breaker, as long as such a person is good at their job and is efficient with what they type (taking advantage of copy/pasting or scripting), it's workable.
Do someone really needs to learn touch typing? It comes naturally when you are using computer daily.
What comes naturally when using a computer daily is usually bad touch typing, i.e. not having to look at your fingers, but typing with only four or maybe six of them using weird, possibly unhealthy, techniques.
I can't stand those supposedly ergonomic keyboards that have been chopped in half, and I didn't understand why anyone would want one until I recalled people learned touch typing.
My naturally acquired typing style involves me holding my hands hands in just about the same angle as those "ergonomic" keyboards, but just using a regular keyboard. Fingers rest on LShift, A, W, D, and K, O, P, Ø with both thumbs on Space.
Should be noted that's on a "105-key" Nordic keyboard with US layout when not writing Norwegian, so the Ø becomes '.
For me this has worked quite fine for over 20 years. I haven't bothered measuring my typing speed because it's never the limiting factor.
My naturally acquired typing style involves me holding my hands hands in just about the same angle as those "ergonomic" keyboards, but just using a regular keyboard. Fingers rest on LShift, A, W, D, and K, O, P, Ø with both thumbs on Space.
Should be noted that's on a "105-key" Nordic keyboard with US layout when not writing Norwegian, so the Ø becomes '.
For me this has worked quite fine for over 20 years. I haven't bothered measuring my typing speed because it's never the limiting factor.
for me it was unhealthy, and hard to fix.
i developed a pretty fast way of typing naturally, using 7 or 8 fingers.
unfortunately it caused wrist strain.
i could not use split ergonomic keyboards because some keys where "mapped" to the wrong hand for that.
i also could not simply learn "proper" touch typing on a qwert layout because my typing habit was too strong.
in the end i learned touch typing on dvorak to fix my wrist issue. maybe after 10 years i'll have unlearned qwert typing enough to go back? (not sure i would, i do prefer dvorak, it just feels silly to type at snail speed on others ppls keyboards).
i developed a pretty fast way of typing naturally, using 7 or 8 fingers.
unfortunately it caused wrist strain.
i could not use split ergonomic keyboards because some keys where "mapped" to the wrong hand for that.
i also could not simply learn "proper" touch typing on a qwert layout because my typing habit was too strong.
in the end i learned touch typing on dvorak to fix my wrist issue. maybe after 10 years i'll have unlearned qwert typing enough to go back? (not sure i would, i do prefer dvorak, it just feels silly to type at snail speed on others ppls keyboards).
Youre right. Im probably lucky to not experience any wrist strain.
Regarding speed - it didnt prevent to be TOP1 at university in typing contest :)
Regarding speed - it didnt prevent to be TOP1 at university in typing contest :)
Depends what you mean by "touch typing". I was typing pretty fast at a young age (young age compared to know) but got my arse handed to me in a speed contest, for fun, at home, by a friend who was a secretary and did learn proper touch typing technique. She was like: "Oh you think you type fast uh?".
So I simply asked her to teach me the proper technique and now, since decades already, I'm touch-typing the proper way.
Having said that: you can be very fast using improper touch-typing technique.
So I simply asked her to teach me the proper technique and now, since decades already, I'm touch-typing the proper way.
Having said that: you can be very fast using improper touch-typing technique.
Yes, someone really do needs to. I used computers daily for 10 years without learning to touch-type, reaching about 24wpm. Then I took a typing course, and in 9 weeks of 5 hours a week I got up to 40wpm, faster than the fastest hunt-and-peck typists. A couple of years later I was at 50wpm, and then I started using IRC and was up to 90 within a few months.
People can practice a skill for a long time, but if they don't focus on improving, they tend to just harden their existing habits.
People can practice a skill for a long time, but if they don't focus on improving, they tend to just harden their existing habits.
No, I don't think they need it, but it helps.
However, it might be a good shortcut for people learning. I was taught at a youngish age to attempt to touch type (box over the keyboard and a tutor program) I fucking hated it, and I was slow, it was dull and painful.
However, Then I had a need for typing, so I got faster and faster, and the initial structure that I got from the rigid teaching, allowed me to learn faster than my peers.
However, it might be a good shortcut for people learning. I was taught at a youngish age to attempt to touch type (box over the keyboard and a tutor program) I fucking hated it, and I was slow, it was dull and painful.
However, Then I had a need for typing, so I got faster and faster, and the initial structure that I got from the rigid teaching, allowed me to learn faster than my peers.
Does it? Most people I know use computers daily, but they can't touch type.
You don't, but boy do I love not focusing on how to type something and just having my thoughts flowing into the screen without me thinking how to do that.
Couldn’t disagree more.
Any activity on a computer is aided by reducing the friction involved in getting what is in your brain into the computer and from the computer back into your brain.
Sure, you can program with punch cards and no display. People did it for years. But having three monitors and a quality keyboard that you’ve mastered will be much more efficient.
Any activity on a computer is aided by reducing the friction involved in getting what is in your brain into the computer and from the computer back into your brain.
Sure, you can program with punch cards and no display. People did it for years. But having three monitors and a quality keyboard that you’ve mastered will be much more efficient.
It's not about clock time, it's about reducing the latency (and friction) between thinking about code and having it written.
I might think for a couple minutes and then come up with ten lines of code, and once they're in my head I want them in my editor now before I lose track of them.
I might think for a couple minutes and then come up with ten lines of code, and once they're in my head I want them in my editor now before I lose track of them.
I don't agree. Touch typing is not about the speed. It is about letting you keep your eyes on the screen while you're typing. It enhances your "headspace" so much, that you don't have to micro-context-switch all the time between the keyboard and the screen.
What about writing that in ancillary to the actual entry of code into the computer?
In the modern workplace there is often quite a bit of communication via a text chat system. A chat where everyone types at 120 WPM is about as fast as the low end of normal conversation speed (120-150 WPM). A chat with several people only doing 40 WPM is much less pleasant.
Email doesn't have the quick back and forth of chat, so slow typing usually won't be directly noticeable to your colleagues, but it can take up a lot of your time.
Someone else already mentioned documentation.
In the modern workplace there is often quite a bit of communication via a text chat system. A chat where everyone types at 120 WPM is about as fast as the low end of normal conversation speed (120-150 WPM). A chat with several people only doing 40 WPM is much less pleasant.
Email doesn't have the quick back and forth of chat, so slow typing usually won't be directly noticeable to your colleagues, but it can take up a lot of your time.
Someone else already mentioned documentation.
Certainly you can be a highly productive programmer if you only hunt and peck, but it's a real handicap; you have to do a lot more of your programming in your head and a lot less in the computer. (Though 40wpm is unheard of, you might get 30.)
I think the best analogue is blindness: it dramatically reduces the bandwidth of the information channel from the computer to you, but there are highly productive blind programmers. Hunt-and-peck programmers are in a similar situation.
I think the best analogue is blindness: it dramatically reduces the bandwidth of the information channel from the computer to you, but there are highly productive blind programmers. Hunt-and-peck programmers are in a similar situation.
One of the best and extremely productive programmers I have known had cerebral palsy. He typed slowly, with one finger, while holding the right hand steady with the left.
Yeah, someone who doesn't care about consistency but can type 120WPM is just going to break your DB faster.
I started with hunt-peck style typing and can now type without looking at the keyboard because of muscle memory. As you mentioned, typing has never seemed like a productivity impediment to me since I don't think I ever had to type at 120WPM when programming.
I like the Zoho strategy of just hiring high school kids and training them - https://m.timesofindia.com/india/this-man-is-also-driving-on...
I've worked with several companies that tried that, in the end once the juniors became experienced and skilled enough, they took offers from other companies with a much higher salary.
The issue is that training/skilling juniors or fresh university graduates costs you money and slows you down (as you need to assign senior engineers/mentors to assist them). It is not unlikely that a newly joined junior employees only costs you money and decreases throughput/value for your company in the first 6-12 months, but you still pay them a (decent) salary. So once you got them skilled, you as a company would probably need to raise the salary to market/senior level to keep them - given the mass amount of recruiter approaches we in IT get. In that case, you could just start off hiring seniors only in the first place.
This is an issue I see everywhere, nobody wants to hire freshly graduated and everybody is only shopping around for seniors.
The issue is that training/skilling juniors or fresh university graduates costs you money and slows you down (as you need to assign senior engineers/mentors to assist them). It is not unlikely that a newly joined junior employees only costs you money and decreases throughput/value for your company in the first 6-12 months, but you still pay them a (decent) salary. So once you got them skilled, you as a company would probably need to raise the salary to market/senior level to keep them - given the mass amount of recruiter approaches we in IT get. In that case, you could just start off hiring seniors only in the first place.
This is an issue I see everywhere, nobody wants to hire freshly graduated and everybody is only shopping around for seniors.
>took offers from other companies with a much higher salary.
Seems like paying at least market rates once they are trained up would solve that problem. Once someone develops a skill, a company can't expect to keep them at intern or junior salary levels.
Seems like paying at least market rates once they are trained up would solve that problem. Once someone develops a skill, a company can't expect to keep them at intern or junior salary levels.
That was exactly my point. But why hire & train a junior dev when there is no company loyalty and the money spent on training is basically lost. Instead management focus on directly hiring seniors. Of course that doesn't work as a whole, because if no one hires juniors there will be a point where there will be no seniors anymore. I also feel little to no pity when those companies complain, that they can't find seniors.
New unethical life hack: make up a failed startup to pad out your résumé to get hired as a senior right out of college.
It’s interesting, many western corporations used to do that, but after decades it eventually was delayed to training during summer breaks or jobs right after university as it is often assumed that even after university they lack any real world knowledge. Cheaper and often a loyal employee, still works but not as mainstream as it used to be.
Dojo’s approach seem to be one step beyond, but it requires even longer commitment to the employee and an asymmetrical market opportunity and economic chance. Definitely will help people and become a win win.
Dojo’s approach seem to be one step beyond, but it requires even longer commitment to the employee and an asymmetrical market opportunity and economic chance. Definitely will help people and become a win win.
It's actually a good practice. I think the intelligence community sort of do this too. On the other hand I think a more practical equivalent is to hire internal candidates who show some interests but didn't get the chance to work on real problems and train them.
I've worked at visual effects studios working on feature films, and in that industry (where proprietary workflows rule) there is always an education department. Even if one used to work there, being gone for a year is enough to make one useless as the workflows change radically, VFX technologies move fast. When I spent time consulting at web shops, I was dismayed at the undocumented and loose workflows, plus no process documentation in sight. I get the sense the web is the worse of modern tech exploitation work practices. Hell, working in video games looks better.
>I get the sense the web is the worse of modern tech exploitation work practices.
Web work is exploitative because... the workflow is undocumented? If it pays well, who cares?
Web work is exploitative because... the workflow is undocumented? If it pays well, who cares?
Undocumented is much worse than your comment seems to realize. Wait until you are in a situation when the only people that know a mission critical procedure quit or die. This can cause entire large companies to crumble, as well as the reason so much confusion exists in development, and is also a reason the industry relies on leet code testing - if you don't know something before being hired, nobody can explain it there because they are all fixing fires or on not-really-understanding auto-pilot.
I feel that this is a perfectly good interview question for a so-called "full stack developer":
> Let's turn to databases now. Can you explain ACID to me?
I'm not saying that the original article on RoyalSloth was wrong, exactly. It has to do with the way the interviewer approaches the question. For instance, for that full-stack developer I would be PERFECTLY satisfied with an answer that went like this:
> Hmm. Well, I don't remember exactly what the letters stand for. Atomic, something something. But look -- some databases, like MongoDB, can sometimes lose data after it has been "committed". Other databases, like DynamoDB, can read just fine from one table but they can't guarantee some basic invariants when reading from multiple tables at once. And some others, PostgreSQL and most "relational databases" fit in this category, jump through hoops (I don't know quite how it works) to make certain that what you see is always consistent.
If I were hiring someone JUST to do front-end web design then this wouldn't be relevant. If it's someone more junior then the above sentiment but without the ability to name specific examples would be fine. But I don't want to hire someone to work on interfacing with the database unless they understand it to at least this level.
If I were hiring for a database specialist I would expect a much more detailed explanation.
> Let's turn to databases now. Can you explain ACID to me?
I'm not saying that the original article on RoyalSloth was wrong, exactly. It has to do with the way the interviewer approaches the question. For instance, for that full-stack developer I would be PERFECTLY satisfied with an answer that went like this:
> Hmm. Well, I don't remember exactly what the letters stand for. Atomic, something something. But look -- some databases, like MongoDB, can sometimes lose data after it has been "committed". Other databases, like DynamoDB, can read just fine from one table but they can't guarantee some basic invariants when reading from multiple tables at once. And some others, PostgreSQL and most "relational databases" fit in this category, jump through hoops (I don't know quite how it works) to make certain that what you see is always consistent.
If I were hiring someone JUST to do front-end web design then this wouldn't be relevant. If it's someone more junior then the above sentiment but without the ability to name specific examples would be fine. But I don't want to hire someone to work on interfacing with the database unless they understand it to at least this level.
If I were hiring for a database specialist I would expect a much more detailed explanation.
> For the majority of programmers out there, something being RESTful means that you can feed it chunks of JSON over HTTP. The minority, who have spent some time reading about it, will complain that it’s a design principle defined in the Roy Fielding’s thesis and it has nothing to do with JSON or HTTP 3. How many of you have actually read his thesis? All of it? No skipping?
I've been railing about this for years. Please stop calling it a REST endpoint. In 95% of the cases it is a custom HTTP endpoint and that's it.
I've been railing about this for years. Please stop calling it a REST endpoint. In 95% of the cases it is a custom HTTP endpoint and that's it.
Probably pick a different hill to die on. The meaning of "REST" in common parlance is now just HTTP+JSON, nothing more. At best, the url might represent a noun.
I'm not going to die on that hill, but I will gleefully call an API "RPC-over-HTTP" as a kind of silly trolling when this style of JSON-over-HTTP RESTful is equated with cool.
And everyone is going to roll their eyes and continue on with their work, as they don't want the 5th discussion about this.
Hence “silly trolling.”
I actually wonder where this came from. It seems like a few popular Spring Boot projects started putting "rest" somewhere in the url path of every http endpoint that returns a serialized data structure in the response body rather than browser-renderable markup, but were they the first to do this?
I try to just call these things an "http API" when talking about it myself, just to mean an endpoint that is intended for consumption by other applications, not by humans, but I'm not gonna fight other people using language the way it works in practice, where words just mean whatever enough people agree they mean, regardless of what the very first user of the word meant.
I try to just call these things an "http API" when talking about it myself, just to mean an endpoint that is intended for consumption by other applications, not by humans, but I'm not gonna fight other people using language the way it works in practice, where words just mean whatever enough people agree they mean, regardless of what the very first user of the word meant.
Knowing fundamentals is important. It is simply part of the professional culture, or part of what they sometimes call a degree of professional “maturity.” A programmer who is not aware of the fundamentals will always find a way to write bad code.
Yes but there is a catch. I've seen some shit from some fundamental lover programmers. The fundamentals are important but not essential to every job, there is hardly a role they can't improve, but they're not guaranteed to give good outcomes. These days is more important to know how to compose high level components in a simple way than optimizing specific algorithms you can learn one from the other tho.
It is... but the book knowledge version of fundamentals and their actual implementation are completely different when you start asking questions.
Even more so when those "fundamentals" are implementation details you don't think about much in your day to day job. Actively recalling knowledge that has become part of your intuition and understanding of the world is hard.
I know enough about many things to do basic implementations safely; everything else is paged out to google. The finer details of database transations? Usually not my job, but I used to know more about them and could relearn that information quickly if needed.
Even more so when those "fundamentals" are implementation details you don't think about much in your day to day job. Actively recalling knowledge that has become part of your intuition and understanding of the world is hard.
I know enough about many things to do basic implementations safely; everything else is paged out to google. The finer details of database transations? Usually not my job, but I used to know more about them and could relearn that information quickly if needed.
A better question would be: ACID (after explaining it's meaning again) has competing concepts. Can you think of any?
I love these "higher order" questions. Every thinking person should have their doubts or opinions about what is thought of as common knowledge, "best practices," etc., be aware of their limitations, know about alternatives; and, in general, despise any sort of dogmatism - especially in fast developing areas such as programming.
Don’t ask questions that can be answered with a web search.
So the onsite interview should be a one day phd program?
This is kind of thinking is why people hire all these leetcode jockeys who can't write working software, and won't stay beyond a few months anyway.
This is kind of thinking is why people hire all these leetcode jockeys who can't write working software, and won't stay beyond a few months anyway.
This is really the least that I can ask for new hires. I don't mind if they want me to repeat what I have taught them because I have been there and know that been taught != fully understand and it takes a few practices to imprint that in one's brain.
But for God's sake please Google before asking any general question. Some questions are literally within the first 5 results of the first page.
Maybe I shouldn't blame them though, sometimes I did that too :/
But for God's sake please Google before asking any general question. Some questions are literally within the first 5 results of the first page.
Maybe I shouldn't blame them though, sometimes I did that too :/
> Don’t ask questions that can be answered with a web search.
These days, that’s equivalent to saying “don’t ask questions about what the day to day job requires one to know”.
Sites like stackoverflow contain answers to problems that people encounter in their day to day jobs, and if you eliminate those as potential questions, then what should you ask? And what is the value in asking some obscure question, other than making the interviewer feel smart and special?
These days, that’s equivalent to saying “don’t ask questions about what the day to day job requires one to know”.
Sites like stackoverflow contain answers to problems that people encounter in their day to day jobs, and if you eliminate those as potential questions, then what should you ask? And what is the value in asking some obscure question, other than making the interviewer feel smart and special?
Here's a question you can't properly answer with a web search for a data engineering role
"I'm emitting 10TB of data a day with this schema {X} - it's produced in Avro. Currently we upload it into S3. But, instead, I want to stream into Apache Druid, but also want to make it available for ad-hoc analysis by BAs.
What approach would you take?"
Can't exactly google an answer to that.
"I'm emitting 10TB of data a day with this schema {X} - it's produced in Avro. Currently we upload it into S3. But, instead, I want to stream into Apache Druid, but also want to make it available for ad-hoc analysis by BAs.
What approach would you take?"
Can't exactly google an answer to that.
Your question turns into a question of "have you worked with Avro, S3, Apache Druid..." which will filter out competent people that don't have experience with the specific list of tools.
No it doesn't.
The keywords there were "streaming" and "also ad-hoc analysis".
Answering those questions requires no knowledge of Avro or Druid, but rather, how to build streaming pipelines, and also make the data available for a query engine.
But that said, if you're in data engineering and are unaware of what Avro is, you're probably not at the level of knowledge I'd expect, and I'm happy to filter you out.
Likewise, if you're unfamiliar with S3, why are you even applying for a data engineering role?
The keywords there were "streaming" and "also ad-hoc analysis".
Answering those questions requires no knowledge of Avro or Druid, but rather, how to build streaming pipelines, and also make the data available for a query engine.
But that said, if you're in data engineering and are unaware of what Avro is, you're probably not at the level of knowledge I'd expect, and I'm happy to filter you out.
Likewise, if you're unfamiliar with S3, why are you even applying for a data engineering role?
But how can I be familiar with S3 without any data engineering work? Or maybe I should put it like this, what exactly does "familiar with S3" mean? Does it mean that I used it to store something (I mean even a non-programmer can do that), or that I should be able to manipulate (i/o) the store using Python with Boto (Again this can be done with a tutorial), or something else?
I believe that "something else" usually indicates some previous data engineering work though, so it's an egg and chicken problem.
I believe that "something else" usually indicates some previous data engineering work though, so it's an egg and chicken problem.
> But how can I be familiar with S3 without any data engineering work?
... you don't need to be in data engineering to use S3.
And can you be familiar with it? Read the docs, write some code.
... you don't need to be in data engineering to use S3.
And can you be familiar with it? Read the docs, write some code.
OK I see what you mean. So it's probably close to the second definition.
I think it's a bit too specific. But for sure someone can relate to similar experience. Something like "For sure I haven't dealt with Druid specifically but in my job I tackled with a similar problem for a different set of tools...".
BTW from my experience we are streaming a lot of data (similar to 10TB but definitely more) produced in JSON then converted to Avro using Spark and finally loaded into Vertica and the analysis pipeline. I have never used Druid but from Googling it is a column based data store which I guess is not too different from Vertica. Is there any specific difficulty you think would be encountered with Druid? Because the wording seems to indicate that Druid is not particularly good with ad-hoc analysis.
BTW from my experience we are streaming a lot of data (similar to 10TB but definitely more) produced in JSON then converted to Avro using Spark and finally loaded into Vertica and the analysis pipeline. I have never used Druid but from Googling it is a column based data store which I guess is not too different from Vertica. Is there any specific difficulty you think would be encountered with Druid? Because the wording seems to indicate that Druid is not particularly good with ad-hoc analysis.
> Because the wording seems to indicate that Druid is not particularly good with ad-hoc analysis
Bingo ;)
I was trying to phrase it like I've heard people do so in interviews.
And tbh, a candidate saying "I'm not familiar with Druid, but it sounds like you can't use it do ad-hoc queries with it? Can you tell me more?" would be a very good sign.
Bingo ;)
I was trying to phrase it like I've heard people do so in interviews.
And tbh, a candidate saying "I'm not familiar with Druid, but it sounds like you can't use it do ad-hoc queries with it? Can you tell me more?" would be a very good sign.
Interesting. I'll look into it as it intrigues me. One thing about DE is that we now have so many tools to choose from :/
This is basically saying "Ive worked on this issue for days/weeks, but please solve this in a few minutes and impress me".
The best ones are the one that take the CV and for each project listed they ask you to say how it was and then some tehnical challenges you encountered. Depending on how you lay out the problems, a competent interviewer will know at what level you are.
Author is not completely wrong, but I don't like this attitude of celebrating ignorance. I agree we shouldn't judge others who happen not to know what we know, but IMO it's likely that knowing this concept at a deconstructed level will be helpful to most devs at some point in the future. The ones who don't know it won't even know it would have helped them, because that's how knowledge works. And that's why it's important to celebrate knowledge, not ignorance.
If you’re writing database applications in a commercial environment then knowing what ACID means is part of your job.
If you don’t know that, for example, debiting one account and crediting another needs to be done inside a transaction - or the converse, what you need to do if you don’t have access to transactions - then you shouldn’t be anywhere near a database.
Fundamentals are actually important. You don’t have to know everything, but you should know the fundamentals of the field you work in.
If you don’t know that, for example, debiting one account and crediting another needs to be done inside a transaction - or the converse, what you need to do if you don’t have access to transactions - then you shouldn’t be anywhere near a database.
Fundamentals are actually important. You don’t have to know everything, but you should know the fundamentals of the field you work in.
The important part you need to know in that case isn't the definition of ACID, but the actual transactional properties of the database you're working with. And even among developers working with databases, it varies how well you actually need to understand transactions.
And I would guess that many developers working with relational databases don't actually understand the more subtle differences between transaction isolation levels. I think asking backend developers to know the basics of transactions is reasonable. But e.g. knowing the exact difference between READ COMMITTED, REPEATABLE READ and SERIALIZABLE (and what your specific database actually implements there) might not be necessary in many cases. Though it certainly is good to be able to spot when you're about to enter territory where these differences might matter.
I agree that a backend developer working with databases should know what transaction are. But I don't really need the definition of ACID, and you can really push this question into details that are not relevant for all developers.
And I would guess that many developers working with relational databases don't actually understand the more subtle differences between transaction isolation levels. I think asking backend developers to know the basics of transactions is reasonable. But e.g. knowing the exact difference between READ COMMITTED, REPEATABLE READ and SERIALIZABLE (and what your specific database actually implements there) might not be necessary in many cases. Though it certainly is good to be able to spot when you're about to enter territory where these differences might matter.
I agree that a backend developer working with databases should know what transaction are. But I don't really need the definition of ACID, and you can really push this question into details that are not relevant for all developers.
I agree to a point. But ACID is part of the nomenclature for relational databases. I feel that if I ask a developer what ACID means, it doesn’t matter to me if they know the specific meaning of each letter in the acronym, but it does matter that they’ve heard of the term and can explain the implications of it.
The point of jargon is to allow concise communication between experts. Knowing the jargon is part of the job.
The point of jargon is to allow concise communication between experts. Knowing the jargon is part of the job.
I dislike asking what acronyms mean, as even I can't always remember the letters, although I know about the necessity of transactions and how to implement them.
Such as "I" - what is it again? "Isolation"? "Transactions are independent". Well--yes.
Such as "I" - what is it again? "Isolation"? "Transactions are independent". Well--yes.
> you should know the fundamentals of the field you work in.
...I think that was the entire point of their post.
> I guess we would be much happier if we dared to admit that one man’s fundamentals are another man’s trivia
...I think that was the entire point of their post.
> I guess we would be much happier if we dared to admit that one man’s fundamentals are another man’s trivia
I don't see the connection between your reply and the linked article's content. A major point is already made right at the start when an ACID definition is quoted from a book - which makes it clear that that term is too nebulous and, in conclusion, that simply regurgitating some definition is not helpful in practice.
On the other hand, if you are actually able to design for all those complex points you may still not have one short definition to quote available, which would not speak against you at all. Not having a short definition for ACID to quote may happen to both real experts as well as people who really don't know anything. Because for real experts that definition is too simplified to be of any use.
I saw similar issues a lot in school and in academia, when having to answer some test questions: When you actually know a lot more you may actually get many answers wrong, because the one asking used a much simplified model (and therefore more inaccurate than yours) as basis for creating their question. For example in biology, there are some key tenets, but when you go deeper you learn about all the additional quirks and exceptions. Using the simplified tenet an answer to a question "can this or that happen" would be never, but for someone who knows a lot more that answer would be wrong. As an expert, you no longer use that simplified model you learned early in school.
Example blog post showing the world is more complex than the simple definition: http://www.bailis.org/blog/when-is-acid-acid-rarely/
That was the point of the article, as shown by this example quote:
> How many similarly vague “fundamentals” we have floating around us? Plenty: RESTful, MVC, SOLID are the first things that come to my mind. If you start a discussion about the meaning behind those terms, you will find yourself in the middle of a food fight where everyone will have a slightly different opinion and no consensus will ever be reached.
All I have top do is read HN whenever such a topic comes up. It usually ends up with a lot of back and forth of people who are all experts but have a slightly different angle, and the initially clear definition falls apart when you read the discussions.
On the other hand, if you are actually able to design for all those complex points you may still not have one short definition to quote available, which would not speak against you at all. Not having a short definition for ACID to quote may happen to both real experts as well as people who really don't know anything. Because for real experts that definition is too simplified to be of any use.
I saw similar issues a lot in school and in academia, when having to answer some test questions: When you actually know a lot more you may actually get many answers wrong, because the one asking used a much simplified model (and therefore more inaccurate than yours) as basis for creating their question. For example in biology, there are some key tenets, but when you go deeper you learn about all the additional quirks and exceptions. Using the simplified tenet an answer to a question "can this or that happen" would be never, but for someone who knows a lot more that answer would be wrong. As an expert, you no longer use that simplified model you learned early in school.
Example blog post showing the world is more complex than the simple definition: http://www.bailis.org/blog/when-is-acid-acid-rarely/
That was the point of the article, as shown by this example quote:
> How many similarly vague “fundamentals” we have floating around us? Plenty: RESTful, MVC, SOLID are the first things that come to my mind. If you start a discussion about the meaning behind those terms, you will find yourself in the middle of a food fight where everyone will have a slightly different opinion and no consensus will ever be reached.
All I have top do is read HN whenever such a topic comes up. It usually ends up with a lot of back and forth of people who are all experts but have a slightly different angle, and the initially clear definition falls apart when you read the discussions.
I think the takeaway from. The article is that if this I'd the case then interviews should drop all the meaningless jargon and just ask a candidate to sketch out a solution to a problem where they would have to think about atomization of DB operations.
In a google interview I explained CAP theorem using the speed of light. The interviewer didn't understand. He clearly wanted me to regurgitate the official definition.
Keen to hear your speed of light explanation of CAP :)
The speed of light puts a hard constraint in the simultaneity of data availability in a network. You can't have data replicated faster than c no matter how fast is your hardware.
The author is talking about hiring interviews like it’s some kind of multiple choice test. Interviews should be more like essays. Of course it is not that important that the interviewee knows the exact definition of ACID but they should certainly know what it is about. Knowing about ACID and the problem space it comes from enables people to think about problems that let to the ACID concept as a solution to said problems. It shows that the interviewee might know enough to look things up and might be capable of selecting the right answer in the reference material/search engine results.
Knowing the exact definition of ACID and nothing else means that somebody can memorize things which is useful but not useful on its own.
You want people that can find a solution and have an somewhat informed opinion about it, not people who can just memorize things.
So asking about ACID is a fine thing, but expecting a text book answer is silly.
Edit: ivanhoe said it better in their post and I changed my phrasing to be less inflammatory. (My point of different hiring mind sets on different continents is not the point.)
So asking about ACID is a fine thing, but expecting a text book answer is silly.
Edit: ivanhoe said it better in their post and I changed my phrasing to be less inflammatory. (My point of different hiring mind sets on different continents is not the point.)
Interviews aren't even like school exams at all. School exams are designed to try to ensure that you learn a certain set of material that was covered in a class. If you missed a bunch of it, that's bad. If at the same time you did independent research in a topic that wasn't covered in the class, it's not the business of an exam to discover that.
School exams are supposed to be fair, in the sense that they only cover the topics covered in class, and only to the level of detail and difficulty covered in the class. They test for weaknesses relative to that standard. When a school exam asks a question, it's because you're supposed to know the answer.
If someone takes the school exam definition of fairness into job interviews, they are going to feel terrified and aggrieved. Interviewers talk to candidates from vastly diverse backgrounds. Other candidates didn't go to your school, maybe didn't go to school in the same country as you, maybe didn't go to college at all. There is no fair, agreed-upon set of standard knowledge. If interviewers restrict themselves to "standard" topics, they will undervalue a lot of people who have unique backgrounds and experience. Not only that, but job interviews can't just measure a candidate's weaknesses against a set standard. It isn't even really a matter of "strengths" and "weaknesses" but wanting to discover a candidate's full capabilities. You don't want to miss a candidate's depth of knowledge in an area simply because you failed to ask any harder questions that would have revealed it. When an interviewer asks a surprising question, it's often not because you're supposed to know but because you might know and they don't want to undervalue you because they didn't ask.
School exams are supposed to be fair, in the sense that they only cover the topics covered in class, and only to the level of detail and difficulty covered in the class. They test for weaknesses relative to that standard. When a school exam asks a question, it's because you're supposed to know the answer.
If someone takes the school exam definition of fairness into job interviews, they are going to feel terrified and aggrieved. Interviewers talk to candidates from vastly diverse backgrounds. Other candidates didn't go to your school, maybe didn't go to school in the same country as you, maybe didn't go to college at all. There is no fair, agreed-upon set of standard knowledge. If interviewers restrict themselves to "standard" topics, they will undervalue a lot of people who have unique backgrounds and experience. Not only that, but job interviews can't just measure a candidate's weaknesses against a set standard. It isn't even really a matter of "strengths" and "weaknesses" but wanting to discover a candidate's full capabilities. You don't want to miss a candidate's depth of knowledge in an area simply because you failed to ask any harder questions that would have revealed it. When an interviewer asks a surprising question, it's often not because you're supposed to know but because you might know and they don't want to undervalue you because they didn't ask.
My take is that too many people are telling others how to interview. It is their money, their company, their job, their life. Let them do it the way they want as long as they follow the legal requirements.
If you think you have a better way, tell us about it without passing judgement on others. Tell us why it works, how great it is, give us some qualitative and quantitative data. Most of all, enjoy your competitive advantage. But for god's sake, quit telling other people they are doing it wrong.
If you think you have a better way, tell us about it without passing judgement on others. Tell us why it works, how great it is, give us some qualitative and quantitative data. Most of all, enjoy your competitive advantage. But for god's sake, quit telling other people they are doing it wrong.
While I don’t disagree, the biggest problem I think is that some or even many interviewers just don’t care or not motivated enough to care whether the candidate can indeed be a great help to the company or the project and approach their interviews in a shallow and formal way.
Then that is a great job for other unmotived people. For motivated people, the poor interview is a great signal to look for a different company.
Yes, you can write some storage solution or yet another database without knowing ACID terms but you will still end up somewhere in the CAP triangle. Using common terms like these makes it easier to communicate with other engineers and makes it more obvious that a lot of things have already been solved and perhaps there’s a library that you can use.
In the same spirit you should not write your own cryptographic algorithm and for example you could develop an API that is not RESTful but it’s more interoperable when you are using standards and prior knowledge.
I personally would not hire a guy to work databases if he insists on not having to learn about ACID.
> you will still end up somewhere in the CAP triangle
Drop P and you'll end up with both consistency and availability, right? Right?
("This is a trade-off we are ready to accept")
Drop P and you'll end up with both consistency and availability, right? Right?
("This is a trade-off we are ready to accept")
I'm somewhat ashamed to say that, when I interview, a lot of people who claim "distributed systems experience" get some variant of a line of questions that ends in the user experience of systems that do not aim for partition-tolerance.
How do you drop P?
That's the point, you can't.
Put quite many people say "you cannot have all three of Conistency, Available, Partition Tolerance", which sounds like you can keep any two / drop one at will.
But the right starting point is: "Partitions will happen. When they do, do you drop A or C?"
Put quite many people say "you cannot have all three of Conistency, Available, Partition Tolerance", which sounds like you can keep any two / drop one at will.
But the right starting point is: "Partitions will happen. When they do, do you drop A or C?"
You just assume network partition is never happening. If calls are synchronous, assume they always reach other side and finish. If async, also assume successful outcome immediately after initiating the call.
This "almost always" works.
This "almost always" works.
> Yet, despite coming up with a distributed archiving system that works in production, I wouldn’t be able to explain from the top of my head what the ACID term means.
You could probably talk about the important properties of interactions with a datastore well enough to please a reasonable interviewer. Why assume they aren't reasonable? I really hate the trope of, "XYZ came up in an interview, and therefore I assume the interviewer is making hiring decisions according to a such-and-such rigid criteria I imagined when I went home afterwards."
(The conversation with "Joe" of course is imaginary, because you wouldn't already be started with interviews if you hadn't talked about how you were going to evaluate candidates. If this conversation happened in real life, the blog post would be about planning for interviews and not going rogue with your interview agenda when you're part of a team process.)
Interviews are designed to figure out where a candidate is with their knowledge, all the way from "they don't know anything about databases" to "they can write a little bit of SQL with help" to "they have 10 years of experience with the database we use for our OLTP, and our expert had a great conversation with them about the technical details of a quirk that bit us in production last week." Depending on the position, these might all be acceptable levels of knowledge for a position, not to mention that companies are often hiring for multiple positions, not all of which have been publicized yet, and are often open to redefining a role for the right candidate.
Every candidate has their own idiosyncratic mixture of strengths and weaknesses. Candidates have a hard time believing it, but a hard or unexpected question is usually probing for strength, not weakness. If interviewers only asked questions that a minimally qualified candidate can answer, they would miss your strengths and undervalue you as a potential hire.
You could probably talk about the important properties of interactions with a datastore well enough to please a reasonable interviewer. Why assume they aren't reasonable? I really hate the trope of, "XYZ came up in an interview, and therefore I assume the interviewer is making hiring decisions according to a such-and-such rigid criteria I imagined when I went home afterwards."
(The conversation with "Joe" of course is imaginary, because you wouldn't already be started with interviews if you hadn't talked about how you were going to evaluate candidates. If this conversation happened in real life, the blog post would be about planning for interviews and not going rogue with your interview agenda when you're part of a team process.)
Interviews are designed to figure out where a candidate is with their knowledge, all the way from "they don't know anything about databases" to "they can write a little bit of SQL with help" to "they have 10 years of experience with the database we use for our OLTP, and our expert had a great conversation with them about the technical details of a quirk that bit us in production last week." Depending on the position, these might all be acceptable levels of knowledge for a position, not to mention that companies are often hiring for multiple positions, not all of which have been publicized yet, and are often open to redefining a role for the right candidate.
Every candidate has their own idiosyncratic mixture of strengths and weaknesses. Candidates have a hard time believing it, but a hard or unexpected question is usually probing for strength, not weakness. If interviewers only asked questions that a minimally qualified candidate can answer, they would miss your strengths and undervalue you as a potential hire.
Once upon a time I was sitting in a chemistry class at Rose-Hulman, and the professor proceeded to use all the formulae we had learned through the trimester to explain how a semiconductor diode worked, then proceeded to explain transistors, from a purely chemical reaction rate basis.
I was in awe... and for a while after that, I thought I understood the basics of electronics.
Last year I learned how tubes work, especially when slightly gassy, and thus ignitrons, etc... turns out I didn't know how all electronics worked.
There is always another layer to learn about. The ARP address layer of the network stack, for instance.
The Unicode layer they added between my days as a Turbo Pascal programmer in MS-DOS, and the present. (Which apparently can be used to hide trojan horses)
Always, always, always another layer.
The best we can do is to try to understand all the layers that definitely affect us, and a layer or two more as insurance (or just because it's cool to know)
I was in awe... and for a while after that, I thought I understood the basics of electronics.
Last year I learned how tubes work, especially when slightly gassy, and thus ignitrons, etc... turns out I didn't know how all electronics worked.
There is always another layer to learn about. The ARP address layer of the network stack, for instance.
The Unicode layer they added between my days as a Turbo Pascal programmer in MS-DOS, and the present. (Which apparently can be used to hide trojan horses)
Always, always, always another layer.
The best we can do is to try to understand all the layers that definitely affect us, and a layer or two more as insurance (or just because it's cool to know)
I think the example is very poor, it sounds like a atraw man argument.
Asking about fundamentals in an interview is, surely, a given? I don't necessarily need everyone to know everything about everything but if I met a dev who didn't know some software principles or a DBA who couldn't explain ACID to at least a basic level, why would I want to employ them? Because they tell me that they just work it out when necessary?
I've cleaned up too much code from people who think "just use an array" or searching through a massive list because they don't know how dictionaries work.
The interview questions sound more than reasonable and then the OP tries to destroy this by saying that not everyone knows all the details of everything which is not what most people would call questions on fundamentals.
Asking about fundamentals in an interview is, surely, a given? I don't necessarily need everyone to know everything about everything but if I met a dev who didn't know some software principles or a DBA who couldn't explain ACID to at least a basic level, why would I want to employ them? Because they tell me that they just work it out when necessary?
I've cleaned up too much code from people who think "just use an array" or searching through a massive list because they don't know how dictionaries work.
The interview questions sound more than reasonable and then the OP tries to destroy this by saying that not everyone knows all the details of everything which is not what most people would call questions on fundamentals.
Yeh, um, no thanks. I've had to deal with too many race conditions, random data loss and week-long-debugging of locks to brush off the "fundamentals" as nice-to-have.
It doesn't matter if you're a frontender or writing a shell script to automate something. If you don't understand atomicity, race condition, locking, etc., not only are you going to shoot yourself in the foot and tear your hair out when shit breaks randomly, but you're going to subject the poor souls who come after you, to unnecessary suffering.
One doesn't get good at these fundamentals by getting a Comp. Sci. degree. All of it can be learnt in a few days. What's important is to keep evaluating your code and design all the time against it to ensure you're not building on crooked foundation.
It doesn't matter if you're a frontender or writing a shell script to automate something. If you don't understand atomicity, race condition, locking, etc., not only are you going to shoot yourself in the foot and tear your hair out when shit breaks randomly, but you're going to subject the poor souls who come after you, to unnecessary suffering.
One doesn't get good at these fundamentals by getting a Comp. Sci. degree. All of it can be learnt in a few days. What's important is to keep evaluating your code and design all the time against it to ensure you're not building on crooked foundation.
In my experience there is not much to know about the ACD parts of ACID, because they're just guaranties that are obvious to want and which will naturally make your life easier.
The I (Isolation) part is the one you'll need to understand whenever you scale up to a point where concurrency starts to become non trivial in your product, mainly because there are multiple levels of isolation, which each provides different guaranties, and you'll need to understand what level your database uses and what it means for your application.
As for the overall point of the article, it's just a question of scale. The more you scale the more you need to cut through the abstraction to understand what the hell is going on. Hire accordingly.
The I (Isolation) part is the one you'll need to understand whenever you scale up to a point where concurrency starts to become non trivial in your product, mainly because there are multiple levels of isolation, which each provides different guaranties, and you'll need to understand what level your database uses and what it means for your application.
As for the overall point of the article, it's just a question of scale. The more you scale the more you need to cut through the abstraction to understand what the hell is going on. Hire accordingly.
They may be obvious to you, but you'd be surprised at how many people out there are doing stuff without optimistic concurrency control, locking or transactions. I mean Mongo only added transactions for updating multiple documents relatively recently. Plus from what I remember they originally missed out the "D" due to only writing to disk intermittently.
I feel like the big issue in the provided example is not so much the fundamentals, but that the "fundamentals" in this case is just buzzword trivia. Maybe a better question would be to ask about atomicity and isolation specifically in the context of a DB?
For example, to me a fundamental for C++ development is knowing the difference between an std::vector and an std::list. Can you really be an effective C++ programmer without knowing that?
Yes the field is vast and lists and vectors can mean different things in different contexts, but you're applying to a C++ development job so I expect you to know some basics about C++ specifically...it doesn't seem that unreasonable to me.
For example, to me a fundamental for C++ development is knowing the difference between an std::vector and an std::list. Can you really be an effective C++ programmer without knowing that?
Yes the field is vast and lists and vectors can mean different things in different contexts, but you're applying to a C++ development job so I expect you to know some basics about C++ specifically...it doesn't seem that unreasonable to me.
The fundamentals are important. In my field of engineering (not software) ignorance of the fundamentals means not knowing how to properly run studies, not being able to accurately make recommendations, not understanding proper cause and effect, not knowing the relevancy of the various settings of the industry standard software...etc etc. It's really hard to find that in new graduates and there is virtually no decent training material. It typically takes 5+ years for just minimum competency, but over a decade before someone begins seeing how everything fits together. Is that common elsewhere?
I would be interested what field you're talking about - could be any rather specialized field, really
Power grid operations. Things are enormously complex (business rules, standards, software stacks, processes, massive data sets, and of course the fundamentals).
Yeah, maybe this is all fields, which is a shame.
Yeah, maybe this is all fields, which is a shame.
The interview style that requires lots of prep studying and leetcode memorization probably does produce the desired result. Workers that will spend non-work time on arbitrary company activities of questionable value.
[deleted]
This is basically because interviewers need to properly advertise the level of depth they want in a role. It's unfair to ask for each engineer to be in the top 5% of engineers everywhere -- not every engineer can be -- but there are roles that don't require such depth.
But interviewing in general is tough (I'd go so far as to say it's a shitshow) and there's lies on both ends -- companies and managers and folks on the inside just want the best of the best -- and candidates just want a chance to get in the door -- the incentive to embellish is too high.
But interviewing in general is tough (I'd go so far as to say it's a shitshow) and there's lies on both ends -- companies and managers and folks on the inside just want the best of the best -- and candidates just want a chance to get in the door -- the incentive to embellish is too high.
Interviews are creative fiction: the applicant pretends to be competent and the employer pretends not to be a toxic workplace environment. I’m increasingly convinced that this is not a solvable problem.
I think this discussion contrasts two opposite types of "fundamentals".
Some "fundamentals" sit at the base of a knowledge pyramid. If an OS developer doesn't know pointers, there's no easy fix for that. You want someone with years of experience building things on top of pointers.
But other "fundamentals" are more like things that are used very commonly and thus one expects that experts would be familiar with them, but they sit on top of knowledge pyramids. It's not hard for experts to pick them up.
I think this lens can help clarify a lot of these discussions!
Some "fundamentals" sit at the base of a knowledge pyramid. If an OS developer doesn't know pointers, there's no easy fix for that. You want someone with years of experience building things on top of pointers.
But other "fundamentals" are more like things that are used very commonly and thus one expects that experts would be familiar with them, but they sit on top of knowledge pyramids. It's not hard for experts to pick them up.
I think this lens can help clarify a lot of these discussions!
My company went public and was acquired, all while doing basic database shit hilariously wrong and just strapping in memory caches everywhere instead of using indexes.
We have two caches and we really just needed to use the dB in a not criminally bad way.
But the market didn't care and the people who did the abuse are now director or architects or manager level engineers at other (good) companies.
There's some reality that knowing this stuff matters but it honestly doesn't for a huge number of tech startups, at least in this market.
We have two caches and we really just needed to use the dB in a not criminally bad way.
But the market didn't care and the people who did the abuse are now director or architects or manager level engineers at other (good) companies.
There's some reality that knowing this stuff matters but it honestly doesn't for a huge number of tech startups, at least in this market.
I have gotten to where is someone asks me to do a linked list or recursive tree search I just end the interview right there. Ask me their strengths or why a balanced tree is useful, awesome. It's fine I hope they have find the right candidate, but as an embedded person I haven't had to write those in decades and if I need to I'll look it up. Sorry I don't have everything a CS grad has fresh in his mind or something that you write daily but most of us don't
I find that surprising. When I'm doing embedded work, explicit linked lists are common. Usually in C, sometimes in assembly language. sometimes even electronics on an FPGA. In many of my projects, rustling up a specialised list or tree based data structure comes up as a side effect of optimising performance, which is often required on smaller devices.
To me non-ACID means you can never use any logical reasoning to have an idea of what there is recorded in your database at any given moment. ACID means you execute a piece of code in a transaction and if something goes wrong it fails altogether returning you to the pre-transaction state. Am I wrong?
Yes, you are wrong. For example, the "I" in ACID stands for isolation, which offers some guarantees regarding concurrently running transactions. In particular, if you have 2 concurrently running transactions A and B, and A finishes before B, and B fails, then the database does not return to the pre-transaction state (if failing transaction B would cause transaction A to be rollbacked, the Isolation principle would be violated).
Furthermore, many real-world databases do not fully provide ACID guarantees, and yet you can reason about the database anyway.
Furthermore, many real-world databases do not fully provide ACID guarantees, and yet you can reason about the database anyway.
"Yet, despite coming up with a distributed archiving system that works in production". The thing is, did he? Really? I know plenty of people who get things into production and by working they mean, doing something that appears like what it's supposed to do. I usually find these are the people who blame everything else for why it's not working. A red flag is usually the phrase, "X isn't ready for prime time". The translation for that is, "I found something that I couldn't figure out and probably will never figure out so you better let me choose something else". These are the people who habitually bounce around from framework to framework. Oddly the more cutting edge it is the better. That way they're on the cutting edge so you need to expect problems, better than everyone else, and because they're better than everyone else you need to listen to them when they tell you, "It's not ready for prime time."
> The thing is, did he? Really?
Well, in the absence of evidence in the contrary, they probably did.
Well, in the absence of evidence in the contrary, they probably did.
Believing everything you read on the internet is true as long as you don't have contrary evidence isn't a good idea. I'm not saying he didn't, I'm saying you don't have any evidence, other than his own assertion, that he did. Considering the article is railing against learning "the fundamentals" his assertion is very self serving and should be taken with a bit of skepticism.
I have 10 years experience and an MS. It's possible I would fail these fundamentals (I have in the past). At my company, we don't talk using big words for tech. This is mostly because we would just end up translating it for the business folks in the room.
Beginners tend to obsess about the precise definitions of terms. Possibly because a lot of teaching is about learning definitions (because it is easy to test) or perhaps because in subject like math the definition really is the meaning.
> not every programmer is dealing with performance issues as part of their job. Some has also specialized in writing GUIs and are more concerned with the user experience and making the user interfaces pretty.
This is a really important point that we don’t talk about enough, because everyone gets uncomfortable and feels judged. But I can’t help myself, and I think there’s a lot more to dissect here.
I think programming is slowly bifurcating. We’re growing into two disciplines.
One discipline requires deep systems knowledge. A CS degree is recommended and if you can’t reverse a binary tree on a white board then you won’t be able to read our code base. Developers in this camp work on rendering engines, Vulcan drivers and schedulers. They spend lunch telling stories about that one time we wrote a game engine in J. Or arguing about type systems.
The second group makes most of the software (measured by lines of code). And almost all the software humans actually interact with. They spend their time working around weird safari quirks, arguing about electron, and enjoying the sense of ease and mastery that comes from staying in a single ecosystem for a decade. Getting better at their job looks like getting better at talking to users and customers, making on point estimates and making sure the team releases on time.
The part that gets all of us in trouble is that the first skill set is (arguably) harder - and thus much more prestigious. Everyone wants to be seen as an electrical engineer, not an electrician. So people write “software engineer” on their resumes. And then cry foul when they get assessed as a member of that first group. “I’ve never needed to know about red black trees in my career. It’s stupid for anyone to ask about it in an interview. Everyone just crams leetcode. JQuery isn’t flashy but it works and I can solve real problems with it. Hire me you fools, and I’ll show you!”. They’re sort of right. We’re just collectively confused about what sort of job they’re being hired for.
I think this confusion hurts everyone. I once wore my fancy engineering hat at a normal consulting gig. I coded up a complex solution to a tricky problem they had and called it a day. After I left, nobody on the team could understand or maintain some of the code I wrote. That caused huge problems for my client after I left. I think they might have regretted bringing me on board at all.
But we can’t even talk about this because we don’t have the language. We treat application developers like they’re just junior, in training versions of deep CS systems engineers. But practicing building kick-arse iOS apps doesn’t make you magically good at data structures. And vice versa - doing performance work doesn’t make you good at working with clients. (Ask me how I know). The majority of professional programmers do plumbing / user facing software. We need to destigmatise this stuff so people can be hired explicitly for these roles, based on the right set of skills.
This is a really important point that we don’t talk about enough, because everyone gets uncomfortable and feels judged. But I can’t help myself, and I think there’s a lot more to dissect here.
I think programming is slowly bifurcating. We’re growing into two disciplines.
One discipline requires deep systems knowledge. A CS degree is recommended and if you can’t reverse a binary tree on a white board then you won’t be able to read our code base. Developers in this camp work on rendering engines, Vulcan drivers and schedulers. They spend lunch telling stories about that one time we wrote a game engine in J. Or arguing about type systems.
The second group makes most of the software (measured by lines of code). And almost all the software humans actually interact with. They spend their time working around weird safari quirks, arguing about electron, and enjoying the sense of ease and mastery that comes from staying in a single ecosystem for a decade. Getting better at their job looks like getting better at talking to users and customers, making on point estimates and making sure the team releases on time.
The part that gets all of us in trouble is that the first skill set is (arguably) harder - and thus much more prestigious. Everyone wants to be seen as an electrical engineer, not an electrician. So people write “software engineer” on their resumes. And then cry foul when they get assessed as a member of that first group. “I’ve never needed to know about red black trees in my career. It’s stupid for anyone to ask about it in an interview. Everyone just crams leetcode. JQuery isn’t flashy but it works and I can solve real problems with it. Hire me you fools, and I’ll show you!”. They’re sort of right. We’re just collectively confused about what sort of job they’re being hired for.
I think this confusion hurts everyone. I once wore my fancy engineering hat at a normal consulting gig. I coded up a complex solution to a tricky problem they had and called it a day. After I left, nobody on the team could understand or maintain some of the code I wrote. That caused huge problems for my client after I left. I think they might have regretted bringing me on board at all.
But we can’t even talk about this because we don’t have the language. We treat application developers like they’re just junior, in training versions of deep CS systems engineers. But practicing building kick-arse iOS apps doesn’t make you magically good at data structures. And vice versa - doing performance work doesn’t make you good at working with clients. (Ask me how I know). The majority of professional programmers do plumbing / user facing software. We need to destigmatise this stuff so people can be hired explicitly for these roles, based on the right set of skills.
>I think programming is slowly bifurcating. We’re growing into two disciplines.
It has always been like this. The distinction used to be: base software developer vs applications developer.
Base software developers went to work on operating systems, databases, web servers, etc.
Application developers worked on business software.
And most decent business software development shops kept a couple of devs with decent knowledge of base development just to go for when the needs got technical.
It has always been like this. The distinction used to be: base software developer vs applications developer.
Base software developers went to work on operating systems, databases, web servers, etc.
Application developers worked on business software.
And most decent business software development shops kept a couple of devs with decent knowledge of base development just to go for when the needs got technical.
The best programmers are better at talking to users and customers, because that is the harder skill. Programming isn’t a solo activity - at the very least the programmer is working with their past and future selves. Which of Linus’s projects have had the bigger impact - Linux or git?
Being able to work with another skilled programmer (be that yourself in the future or someone you're working on a kernel with) is not the same kind of skillset as being able to talk to an end user about what the pain points are in their day to day work and trying to actually sympathise with them and understand what problems they're encountering without making the rookie mistake of taking everything the user said at face value and designing a system around it.
These are disparate skillsets and both of them are difficult to master. Claiming one is harder than the other is really quite shallow. Some people are going to naturally have an easier time picking up one skillset while other will have a naturally easier time picking up the other skillset. I think rather than trying to pretend like people should stop focusing on hard skills and start focusing on soft skills, we could realise for a moment that it does nobody any good to take people who are good at one task and insist that they must retrain for another when there are already plenty of people who are good at the other task.
It would make more sense to get these groups of people to learn just enough of the opposite person's skillset so that they can effectively communicate and as a result form a well functioning two-sided system where one group of people talks to users and analyses their requirements and the other group of people actually works off of these more concrete better defined requirements to actually implement the software.
If you think that "soft skills" are harder than "hard skills" it's probably because they're harder for you, that doesn't mean they're harder for everyone, and instead of pretending like this is some kind of major flaw in your own professional development, you could treat it as a gap which could be filled by people who have the opposite view of difficulty.
These are disparate skillsets and both of them are difficult to master. Claiming one is harder than the other is really quite shallow. Some people are going to naturally have an easier time picking up one skillset while other will have a naturally easier time picking up the other skillset. I think rather than trying to pretend like people should stop focusing on hard skills and start focusing on soft skills, we could realise for a moment that it does nobody any good to take people who are good at one task and insist that they must retrain for another when there are already plenty of people who are good at the other task.
It would make more sense to get these groups of people to learn just enough of the opposite person's skillset so that they can effectively communicate and as a result form a well functioning two-sided system where one group of people talks to users and analyses their requirements and the other group of people actually works off of these more concrete better defined requirements to actually implement the software.
If you think that "soft skills" are harder than "hard skills" it's probably because they're harder for you, that doesn't mean they're harder for everyone, and instead of pretending like this is some kind of major flaw in your own professional development, you could treat it as a gap which could be filled by people who have the opposite view of difficulty.
> The best programmers are better at talking to users and customers, because that is the harder skill.
Uh, what? Most humans find social skills much easier to learn than crunchy engineering skills. Good social skills are way more common in society than, say, deep knowledge of algorithms and data structures.
But it’s not a competition. As you say, programming is rarely a solo activity. Our projects can fail due to a lack of teamwork. Or a lack of technical skill. Or a lack of empathy with our users. Or a lack of leadership. Or a dozen other things.
We need (and need to value, depending on the project) all of this stuff.
Uh, what? Most humans find social skills much easier to learn than crunchy engineering skills. Good social skills are way more common in society than, say, deep knowledge of algorithms and data structures.
But it’s not a competition. As you say, programming is rarely a solo activity. Our projects can fail due to a lack of teamwork. Or a lack of technical skill. Or a lack of empathy with our users. Or a lack of leadership. Or a dozen other things.
We need (and need to value, depending on the project) all of this stuff.
> How many similarly vague “fundamentals” we have floating around us? Plenty: RESTful
Hell yeah, I have hardly ever seen a correct REST API in the wild. People mostly consider any non-SOAP HTTP API RESTful.
Hell yeah, I have hardly ever seen a correct REST API in the wild. People mostly consider any non-SOAP HTTP API RESTful.
I once failed a job interview because the Fortran complier they used handled errors differently than the one I was used to, so they assumed I didn't know what I was talking about.
it’s true that database are lying about ACID and you only get true acid if you change database isolation level from default to “serializable” but on most database (that don’t use mvcc) you would get horrible performance.
I still think any programmer using any database should know what “data consistency” you sacrificed for using “Read commited” instead of “serialisable” and how hacker can use it to steal million of dollar of your e-commerce company
I still think any programmer using any database should know what “data consistency” you sacrificed for using “Read commited” instead of “serialisable” and how hacker can use it to steal million of dollar of your e-commerce company
If you know more about the fundamentals than competing job applicants then you increase your chance of getting the job.
All you need to know is the `if` statement. That's all.
ACID is the kind of fundamental which can make the difference between easily constructing a reliable system and constructing an unreliable system with great difficulty. It's a mistake to class it with the MVC pattern (now diluted past all recognition) and SOLID just because they're all acronyms and your source of information about ACID was a shitty Wild Hog book by someone who wasn't clear about what it was either.
You can build bridges that don't fall down without knowing the difference between stress and strain; you can build an audio amplifier without knowing the difference between voltage and current; and you can build a database-backed application without knowing what ACID means. But you shouldn't, except maybe as a learning exercise, because it's an enormous amount of effort to produce mediocre results. Understanding ACID allows you to treat a database as "a black box with an API" with enormously better results.
I have no patience for the kind of glorification of ignorance in this post. Perhaps it's true that "the majority of programmers" think "something being RESTful means that you can feed it chunks of JSON over HTTP." The majority of audiophiles might think that oxygen-free copper will improve their sound quality, the majority of computer users might think that Google built the Internet, and the majority of people don't speak English. That has no bearing on the knowledge needed to actually build a working hi-fi system, debug an internet connection problem, or successfully converse in English, and neither does popular ignorance about REST (or ACID) have any bearing on the knowledge needed to successfully design a distributed system that works reliably in the face of constant partial failures.
(Ivanhoe's post at https://news.ycombinator.com/item?id=29065438 is an excellent explanation of what can happen when you lack the understanding of these fundamentals but mistakenly think you can use the database as a black box anyway: "I lost a lot of data when traffic suddenly jumped, and since those were affiliate clicks lots of people got very pissed about not getting paid, and it almost ruined both my client's business as well as my own, as it was our core client.")
If over 40 years you spend 83000 hours working, and you spend 10% of them on programming, that's 8300 hours. (If your job title is "software engineer" you probably spend more like 50% of them on programming, but maybe you're a web designer, a database administrator, a sysadmin, a mechanical engineer, or a biologist, so programming is a relatively minor part of your work.) The optimal fraction of that time spent on getting to know the fundamentals --- real fundamentals like ACID, not fake fundamentals like SOLID, though this presumes that you have someone to guide you who knows the difference --- is surely not 100%, because then you'll never get any work done. But it's not 0% either; you won't get much done that way either. It's probably in the range of 20% to 60%, which would be 1700-5000 hours. Learning in your bones what ACID is might take you 5 hours spread over a month or two, so it only takes up 0.3%-0.1% of that time.
You can build bridges that don't fall down without knowing the difference between stress and strain; you can build an audio amplifier without knowing the difference between voltage and current; and you can build a database-backed application without knowing what ACID means. But you shouldn't, except maybe as a learning exercise, because it's an enormous amount of effort to produce mediocre results. Understanding ACID allows you to treat a database as "a black box with an API" with enormously better results.
I have no patience for the kind of glorification of ignorance in this post. Perhaps it's true that "the majority of programmers" think "something being RESTful means that you can feed it chunks of JSON over HTTP." The majority of audiophiles might think that oxygen-free copper will improve their sound quality, the majority of computer users might think that Google built the Internet, and the majority of people don't speak English. That has no bearing on the knowledge needed to actually build a working hi-fi system, debug an internet connection problem, or successfully converse in English, and neither does popular ignorance about REST (or ACID) have any bearing on the knowledge needed to successfully design a distributed system that works reliably in the face of constant partial failures.
(Ivanhoe's post at https://news.ycombinator.com/item?id=29065438 is an excellent explanation of what can happen when you lack the understanding of these fundamentals but mistakenly think you can use the database as a black box anyway: "I lost a lot of data when traffic suddenly jumped, and since those were affiliate clicks lots of people got very pissed about not getting paid, and it almost ruined both my client's business as well as my own, as it was our core client.")
If over 40 years you spend 83000 hours working, and you spend 10% of them on programming, that's 8300 hours. (If your job title is "software engineer" you probably spend more like 50% of them on programming, but maybe you're a web designer, a database administrator, a sysadmin, a mechanical engineer, or a biologist, so programming is a relatively minor part of your work.) The optimal fraction of that time spent on getting to know the fundamentals --- real fundamentals like ACID, not fake fundamentals like SOLID, though this presumes that you have someone to guide you who knows the difference --- is surely not 100%, because then you'll never get any work done. But it's not 0% either; you won't get much done that way either. It's probably in the range of 20% to 60%, which would be 1700-5000 hours. Learning in your bones what ACID is might take you 5 hours spread over a month or two, so it only takes up 0.3%-0.1% of that time.
Because of such attitude of the previous dev team my client ended up with DB integrity ruined. Previous guys somehow didn't know they should use transactions when updating/deleting stuff in the DB, because hey, it's just API you call, who cares of mambo-jumbo happening behind the curtains, right?
I myself had a similar fuckup with MongoDB a decade ago, because I used it "because it's fast", but failed to read the small letters, and wasn't aware that it achieves that speed by buffering the data for async save, and so has no guarantees it will actually be saved at all. So I lost a lot of data when traffic suddenly jumped, and since those were affiliate clicks lots of people got very pissed about not getting paid, and it almost ruined both my client's business as well as my own, as it was our core client.
Lack of understanding of (or even caring to know about) DB internals is also why like 90% of projects I've seen has wrong indices, or often no even a single index set at all. Because, hey, it's someone else's job to think about it, but in reality budgets are limited and team don't have a dedicated DBA or even a capable devops person to notice it, and it just ends up in the production like that.
So, in summary: no you don't need to know every in-and-out of every technology out there, no one can do that, but there's a valid requirement that one has to be familiar to those that they actually use - at least those that can seriously bite one's ass and cost a lot of money. And all DB related stuff is very much like that, that is probably the most critical part of the whole system, so no, it's definitely not a black box.