Systems Design for Advanced Beginners(robertheaton.com)
robertheaton.com
Systems Design for Advanced Beginners
https://robertheaton.com/2020/04/06/systems-design-for-advanced-beginners/
114 comments
This is interesting advice because every company I've come across that has been successful enough to get past the early startup phase has by and large ignored all of this.
The companies I've seen succeed were 100% focused on shipping their product to customers. Not 90% focused on customers and 10% focused on code quality, but 100% focused. They'd rather have to spend 30 engineer-days a few years from now fixing an issue if they get to that point than spend 3 hours getting it right upfront.
As an engineer that goes against every instinct I have, it really seems like spending a couple hours upfront must be a better use of time. It seems like it should be possible to spend 10% of your time setting yourself up well for the future, that's still just a rounding error of your time. And then if you do survive another few years, you'll have a huge leg up on other series B or C stage competitors if you're not hindered by a lot of tech debt at that point.
But from a capitalist perspective, it's probably not so crazy. If you are working with a $200,000 seed round in the beginning, and an engineer costs $80,000 a year, 3 hours of their time costs $115 which is 0.06% of your funds. And more importantly, that $200k is maybe enough for a year of runway, so 3 hours is 0.14% of the time you have to live given a 40-hour a week year (or 0.07% of an 80 hour a week year). Every bit of that starts to add up. Whereas by the time you're a later-stage company and you've raised, say, $40 million dollars and are paying engineers $150k, 30 engineer-days of work is $17,300 but that's only 0.04% of the money you've raised plus your runway is now approaching infinity if you're close to profitable.
I'm still kind of playing devil's advocate here, my instinct really wants to believe that a better balance than what I've seen is possible. One huge missing factor is that people have a strong tendency to ignore spread out costs like the time wasted fixing bugs that pop up later whereas the upfront cost of writing a bunch of tests is more visible. But it has been interesting for me to consider that maybe most founders really are acting pretty reasonably, even though it originally seems pretty careless and lazy to let your startup build up a ton of tech debt early on.
The companies I've seen succeed were 100% focused on shipping their product to customers. Not 90% focused on customers and 10% focused on code quality, but 100% focused. They'd rather have to spend 30 engineer-days a few years from now fixing an issue if they get to that point than spend 3 hours getting it right upfront.
As an engineer that goes against every instinct I have, it really seems like spending a couple hours upfront must be a better use of time. It seems like it should be possible to spend 10% of your time setting yourself up well for the future, that's still just a rounding error of your time. And then if you do survive another few years, you'll have a huge leg up on other series B or C stage competitors if you're not hindered by a lot of tech debt at that point.
But from a capitalist perspective, it's probably not so crazy. If you are working with a $200,000 seed round in the beginning, and an engineer costs $80,000 a year, 3 hours of their time costs $115 which is 0.06% of your funds. And more importantly, that $200k is maybe enough for a year of runway, so 3 hours is 0.14% of the time you have to live given a 40-hour a week year (or 0.07% of an 80 hour a week year). Every bit of that starts to add up. Whereas by the time you're a later-stage company and you've raised, say, $40 million dollars and are paying engineers $150k, 30 engineer-days of work is $17,300 but that's only 0.04% of the money you've raised plus your runway is now approaching infinity if you're close to profitable.
I'm still kind of playing devil's advocate here, my instinct really wants to believe that a better balance than what I've seen is possible. One huge missing factor is that people have a strong tendency to ignore spread out costs like the time wasted fixing bugs that pop up later whereas the upfront cost of writing a bunch of tests is more visible. But it has been interesting for me to consider that maybe most founders really are acting pretty reasonably, even though it originally seems pretty careless and lazy to let your startup build up a ton of tech debt early on.
Engineering is a multiplier in this case - so early investments, not necessarily in code beauty, but in picking the right architecture for the job, pays back compounding dividends and reduces funding needed. Whatsapp had a handful of engineers able to handle millions of users. Funding is not promised, its debt. Use every lever you've got and you'll need less of it.
There are compounding gains from making "the right choice all else equal." This is one of those semi-mythical 10x engineer super-powers. Or it might just be experience. But technical choices need to be made and making slightly better ones will pay off even in the short term. The technical changes should just optimize for rapid change rather than stability or scale.
[deleted]
How do you prototype? What do you look for while prototyping lets say, nginx or haproxy? You certainly can't simulate real world traffic on a toy application. So how do you decide which one to pick? I am beginner in this field.
You can simulate traffic with Blitz or Bees With MachineGuns or many other tools. You can use your own prototypes for a few days. You can create fake words and test how your approaches rank against each other on google (and keep tracking that over time in case algorithms change). There is a lot of inherent reward in finding creative ways to break your own stuff and the learning is unparalleled.
Thought I'd post this here since I imagine there's a strong cross-over of interest:
If you'd like a guided video code-tour of how all the pieces of a production web-app fit together, I publish detailed weekly screencasts showcasing the code, systems, and architecture behind Oxbridge Notes, the business that's supported me for the past decade.
So far I've covered:
— software dependency vetting
— data integrity systems (constraints, foreign keys, transactions etc.)
— integration testing systems
— trade-offs in software quality between customer-facing and admin areas
— softer stuff, like designing for SEO (marketing ease is a critical part of any system I design, as an indie-hacker)
https://www.semicolonandsons.com/series/Inside-The-Muse
If you'd like a guided video code-tour of how all the pieces of a production web-app fit together, I publish detailed weekly screencasts showcasing the code, systems, and architecture behind Oxbridge Notes, the business that's supported me for the past decade.
So far I've covered:
— software dependency vetting
— data integrity systems (constraints, foreign keys, transactions etc.)
— integration testing systems
— trade-offs in software quality between customer-facing and admin areas
— softer stuff, like designing for SEO (marketing ease is a critical part of any system I design, as an indie-hacker)
https://www.semicolonandsons.com/series/Inside-The-Muse
Hey Jack,
Just a heads up - I tried to sign up for your mailing list but kept seeing "ERROR: Did you forget to type your email?" even after typing the email. I've tried a few different emails and also tried viewing the page in incognito mode. Hope you get this fixed soon, as I'd really love to get more of your content!
Just a heads up - I tried to sign up for your mailing list but kept seeing "ERROR: Did you forget to type your email?" even after typing the email. I've tried a few different emails and also tried viewing the page in incognito mode. Hope you get this fixed soon, as I'd really love to get more of your content!
I get the same error :/
Also, it'd be great to be able to turn off the mailing list sign up on the videos. I'm already technically subscribed via RSS and it'd be great if the videos play through without intervention
Also, it'd be great to be able to turn off the mailing list sign up on the videos. I'm already technically subscribed via RSS and it'd be great if the videos play through without intervention
Thanks for the head's up! I checked the logs and it was just a case of an incorrect error message. Fixed now.
Your site appears blocked on Vodafone UK's mobile network. I get a connection interrupted error, the same error I get if I try to access a site I know is blocked (such as archive.org). If I use a VPN I can load the page, even if the VPN connection terminates in the UK.
Interesting. Thanks.
They have this diagram:
+-----------+ +--------------+ +-----------------+
|Web Browser| |Smartphone App| |Client Libraries/|
+-----+-----+ +------+-------+ |Other API code |
| | +-------+---------+
| v |
| +-----+------+ |
+--------->+ Steveslist +<-----------+
| Servers |
+------------+
+-----------+ +-----------------+ +--------------+
|Web Browser|-->|Client Libraries/|<--|Smartphone App|
+-----------+ |Other API code | +--------------+
+-------+---------+
|
v
+-----+------+
| Steveslist |
| Servers |
+------------+Yes, I think the second diagram that you showed is the preferred way of doing things, since using or "dogfooding" the client libraries will ensure they are always tested and correct, plus you'll get immediate feedback from the other developers who work on the apps as well.
In this day and age, your client libraries should be generated automatically. And of course used in your apps/webpage to save time.
See openapi/swagger for good examples how client generation works
See openapi/swagger for good examples how client generation works
I've been using openapi/swagger for generating my clients and honestly think it creates a mess of files and manually keeping clients up to date is not a huge task. I'm not sure that I buy into "your client libraries should be generated automatically" as a blanket statement.
Granted, APIs aren't updated often, but when they are, generated clients can save a lot of time. I can update a generated client in half an hour and be very confident of its correctness, whereas a manually constructed client would take hours to update and test.
Of course, the generated client is not very convenient, being a 1-to-1 mapping with the API. You build a high-level client with the more common operations on top of the generated one.
It's possible to do this in a way that gets you the best of both approaches, with some up-front planning.
Of course, the generated client is not very convenient, being a 1-to-1 mapping with the API. You build a high-level client with the more common operations on top of the generated one.
It's possible to do this in a way that gets you the best of both approaches, with some up-front planning.
Another benefit would be easier testing - you can easily auto-test the logic and separate the testing of the UI.
Nicely wrapped-up public APIs with a pleasant presentation & small learning curve require tradeoffs. Why force your own app & website to pay those costs?
You dont "force your site to pay these costs", you design your clients as consumers from day one and get consumable external APIs for free.
This is a common "holy grail" that I've never seen any company pull off successfully. Like others have said you rarely have the ability to build a simple-yet-powerful API that perfectly services both your company proprietary apps and 3rd party clients. The requirements are almost always at odds with each other.
It certainly takes a huge amount of discipline, which is wholly absent in most service companies.
Hiding the costs, minimising the costs or "integrating them well with your platform" do not erase the costs.
Do most companies need an external API?
3/4 places I've worked have an external API. It might not be "public" but often enterprise customers want to pull their data for their own mangling.
I think at the beginning you won't have a client API at the beginning.
That stuff is only done at the end unless you're chargin for that at the beginning.
If you're not B2B positioned, you probably won't have a client library for a while.
That stuff is only done at the end unless you're chargin for that at the beginning.
If you're not B2B positioned, you probably won't have a client library for a while.
Wouldn't the issue with client libraries be tight-coupling and having to maintain n libraries instead of one generic server API endpoint?
Yes
Different ergonomics of each of those environments, at least in the ideal case for each combined with the server API being a clean enough API boundary if I had to guess.
Nice tutorial! One small nitpick:
> Calculating a hash value from an input is computationally very easy, but reversing the transformation and recovering the original input from its hash value takes so much time and computing power that it is, practically-speaking, impossible.
The above is true for encryption but not for hash codes. Recovering the original input from a hash code is not just practically impossible; it's provably impossible -- even with an infinite amount of computing power -- because in general a hash code contains less information than the original text.
> Calculating a hash value from an input is computationally very easy, but reversing the transformation and recovering the original input from its hash value takes so much time and computing power that it is, practically-speaking, impossible.
The above is true for encryption but not for hash codes. Recovering the original input from a hash code is not just practically impossible; it's provably impossible -- even with an infinite amount of computing power -- because in general a hash code contains less information than the original text.
Actually, it might be occasionally possible. Not perfectly, of course. If the input space is limited and you have other constraints on it, then you might be able to find the original text. For example, let's say your input is a 280 character long tweet, the hash is only 16 bytes long.
If you find that "I AM A STABLE GENIUS" is one of the inputs that hashes to the given hash value, then you can be reasonably sure that this was the original text, given that most of the other inputs with the same hash won't be meaningful English sentences.
I'd say the count of believable English tweets would be many orders of magnitude greater the count of 16 byte hashes, so in general we can't just find a believable tweet and declare that we found the plaintext. There would be many possible collision.
The phrase you mentioned is notable (with a slight edit) - so I suppose you mean you could index phrases by notability, then it may be tractable find tweets of notable phrases, since the number of notable phrases is relatively low.
The phrase you mentioned is notable (with a slight edit) - so I suppose you mean you could index phrases by notability, then it may be tractable find tweets of notable phrases, since the number of notable phrases is relatively low.
I think the author means recovering the hash by using dictionary attacks
You’d still have hash collisions
Maybe not enough to matter though. If the input space is sufficiently smaller than the hash space (e.g., you aren't hashing arbitrary strings, but instead just arbitrary English words) then the probability of any hash collision that also lies in the realm of interest is vanishingly small.
True, but that'd be very esoteric - a string input field that only accepts values from a dictionary. Although also if the input space was only these values but the input size was infinite you'd have a similar issue :P
It'd be esoteric for a system to enforce only containing values from a dictionary, but every sentence ever spoken by anyone alive fits in a 64-bit space by a wide margin. For real-world input you could still reverse most passwords uniquely given enough time.
It's actually kind of a fun problem -- the fewer bits you have in your hash the easier it is to find _any_ collision that gives access to the current system, but the harder it is to uniquely reverse the hash into a plausible password for stuffing into other systems.
It's actually kind of a fun problem -- the fewer bits you have in your hash the easier it is to find _any_ collision that gives access to the current system, but the harder it is to uniquely reverse the hash into a plausible password for stuffing into other systems.
[deleted]
I think the author is just using imprecise language for a preimage attack.[0]
[0] https://en.wikipedia.org/wiki/Preimage_attack
[0] https://en.wikipedia.org/wiki/Preimage_attack
Information is sometimes destroyed, but we can't count on it. An algorithm like HMAC might be used to authenticate a short message. Or even if the message is long, it might also be known, so perhaps the only unknown input (the key) is short. HMAC still provides security in this case, but it's not provable-because-information-is-destroyed in the sense you're describing.
215D70DA68DBDC2217D13916E40F89B87DDC3DAACA99892C943E0DCC76C8C1AF68097C86CC4CB53F15BD6C3870418A1E3577A076611A5104C6D4FC33939A21FB
I said "in general".
> in general a hash code contains less information
That is true, but has nothing to do with your ability to use a rainbow table or dictionary attack on my hashed statement.
The specific reason why you can reverse my hash is because it's unsalted. The time-complexity of my example hash is somewhat low.
'you are wrong' is 13 characters that spans the lowercase alphabet with spaces, in all english words that can be found in a dictionary. The actual key-space is 13^10 because of the limited characters I used, but when creating a rainbow-table it would probably take a key-space of 13^27 because you don't know the specific characters I'm using. (There are 13 characters in my statement, then there are 26 lower-case letters in the alphabet - plus space - which makes 27... 13^27)
Now, if I followed general op-sec suggestions I would have added a salt to my statement, so the process looks like SHA512('you are wrong' + '7aomAxgeVjAvyDXGrdmNJNKuuiumYbkG') which gives you the possible key-space of 45^63. Also, the salt should be something not found in a dictionary.
2812CDA67BF0EA2D5FE125C2637466FA9A81C12AE9D101771C581DB87EBEB05D7724C9DCFBC6F905B99F9737948543EC64CAC5D89C785125DDA2E3297214CC58
On the upper-side of the estimate there are 10^82 atoms in the universe. There are about 14^103 possible solutions to that salted hash above, with many possible collisions. Good luck with creating a database that needs more rows than available atoms, or waiting for a brute-force to complete on that hash, then sorting through the collisions. This is the "impossible" reversible hash you are talking about.
If you are interested in this topic and would like to do this in your code: use the HMAC function which is built for this, or specifically for passwords use bcrypt/blowfish because that algorithm is designed to run slowly (brute force protection).
That is true, but has nothing to do with your ability to use a rainbow table or dictionary attack on my hashed statement.
The specific reason why you can reverse my hash is because it's unsalted. The time-complexity of my example hash is somewhat low.
'you are wrong' is 13 characters that spans the lowercase alphabet with spaces, in all english words that can be found in a dictionary. The actual key-space is 13^10 because of the limited characters I used, but when creating a rainbow-table it would probably take a key-space of 13^27 because you don't know the specific characters I'm using. (There are 13 characters in my statement, then there are 26 lower-case letters in the alphabet - plus space - which makes 27... 13^27)
Now, if I followed general op-sec suggestions I would have added a salt to my statement, so the process looks like SHA512('you are wrong' + '7aomAxgeVjAvyDXGrdmNJNKuuiumYbkG') which gives you the possible key-space of 45^63. Also, the salt should be something not found in a dictionary.
2812CDA67BF0EA2D5FE125C2637466FA9A81C12AE9D101771C581DB87EBEB05D7724C9DCFBC6F905B99F9737948543EC64CAC5D89C785125DDA2E3297214CC58
On the upper-side of the estimate there are 10^82 atoms in the universe. There are about 14^103 possible solutions to that salted hash above, with many possible collisions. Good luck with creating a database that needs more rows than available atoms, or waiting for a brute-force to complete on that hash, then sorting through the collisions. This is the "impossible" reversible hash you are talking about.
If you are interested in this topic and would like to do this in your code: use the HMAC function which is built for this, or specifically for passwords use bcrypt/blowfish because that algorithm is designed to run slowly (brute force protection).
> How do they store their data?
SQL.
> How do their different applications talk to each other?
Proprietary APIs.
> How do they scale their systems to work for millions of users?
T H E C L O U D
> How do they keep them secure?
They just... don't.
> How do they make sure nothing goes wrong?
They just... don't.
> What are APIs, webhooks and client libraries, when you really get down to it?
Easily outsourced to India.
SQL.
> How do their different applications talk to each other?
Proprietary APIs.
> How do they scale their systems to work for millions of users?
T H E C L O U D
> How do they keep them secure?
They just... don't.
> How do they make sure nothing goes wrong?
They just... don't.
> What are APIs, webhooks and client libraries, when you really get down to it?
Easily outsourced to India.
That last point is so painful and true. I work with a product that has dozens if not hundreds of “connectors” to interface with other systems and you can tell that most of them were farmed out to the lowest bidder.
I have been doing a greater part of the things elaborated in the post as a solo founder/freelance software person. Albeit the scale of things I need to handle are relatively small.
What kind of a job profile should I be looking at if I am in a position where I absolutely need to have one?
I don’t slot particularly well in any one thing I feel, neither a great developer nor a great systems person. And I did do a bunch of recruiting and tech consulting too:
What kind of a job profile should I be looking at if I am in a position where I absolutely need to have one?
I don’t slot particularly well in any one thing I feel, neither a great developer nor a great systems person. And I did do a bunch of recruiting and tech consulting too:
Have you ever considered working technical support? As a lifelong generalist programmer, I never imagined I would work in a support role beyond what every programmer at a startup is expected to do, but I was persuaded to cross over by someone I greatly respect. That support team was the most talented team I ever worked with in almost twenty-five years in tech, and working in that role grew many different skills both in and out of tech in ways I never imagined. Hacking broken production environments back online and writing and pushing bugfixes is exhilarating, and it was the single most valuable experience of my career.
I once read about devs that get bored with long projects and thrill at starting new things, and the other kind that doesn't want to take new projects until the one that she/he is working on is "finished".
Maybe you are the first kind and that's why you are enjoying the unexpected situations you face giving technical support.
Never considered support role to be honest. Maybe it feels a little inferior after having those degrees I guess but that’s my own bias. I did think of like a product management role at startups but they all expect you to have pedigree or previous work ex in similar role. Catch-22!
Yeah, that's exactly what I thought, too. It is just one of those weird cultural artifacts that when someone says "tech support" we automatically conjure this mental image which is both not really technical and not really flattering. It was definitely one of those life moments when I had to sit down and talk myself out of a baseless prejudice.
Also, the sibling commenter mentions one of the big reasons I found so much joy in it. This kind of hardcore technical support is in many respects closely related to working on greenfield projects. In my experience, the fun stuff usually ends up with some unfun caveats like company bureaucracy, office politics, or having the tech stack chosen beforehand, this kind of thing. But there's an interesting inversion when a company's production environment is broken and you're the one that can fix it. Obstacles magically disappear and you have a tremendous amount of freedom to do your job, with effectively two different companies doing what they can to enable your work, because the only thing everyone cares about is getting it working again.
You do have to be able to walk the walk -- and I cannot stress enough that it takes a certain type, and just knowing your shit isn't going to cut it -- and you also have to get some personal enjoyment out of chaotic environments, and be able to talk people out of a tree sometimes. But you'll never run out of new and interesting problems to solve, you'll be testing your mettle far more frequently than your peers, and you'll be expanding your professional network every week about as much as everybody else does once or twice a year when they go to a convention. It's the closest thing the tech industry has to a firefighter or superhero or something.
Also, the sibling commenter mentions one of the big reasons I found so much joy in it. This kind of hardcore technical support is in many respects closely related to working on greenfield projects. In my experience, the fun stuff usually ends up with some unfun caveats like company bureaucracy, office politics, or having the tech stack chosen beforehand, this kind of thing. But there's an interesting inversion when a company's production environment is broken and you're the one that can fix it. Obstacles magically disappear and you have a tremendous amount of freedom to do your job, with effectively two different companies doing what they can to enable your work, because the only thing everyone cares about is getting it working again.
You do have to be able to walk the walk -- and I cannot stress enough that it takes a certain type, and just knowing your shit isn't going to cut it -- and you also have to get some personal enjoyment out of chaotic environments, and be able to talk people out of a tree sometimes. But you'll never run out of new and interesting problems to solve, you'll be testing your mettle far more frequently than your peers, and you'll be expanding your professional network every week about as much as everybody else does once or twice a year when they go to a convention. It's the closest thing the tech industry has to a firefighter or superhero or something.
Maybe infosec or cyber security would be similar? I'm a fullstack turned mostly FE developer these days, but I'm very interested in security and it seems like there is a lot of demand out there for that skill set and plenty of opportunity. It seems like it would push a lot of the same buttons that you mentioned (fixing broken systems, lots of autonomy, working with smart folks). I tend to get bored with long-term projects unless there are still interesting problems to be solved so working on lots of smaller projects sounds pretty appealing, and there's seemingly no end to new systems/devices/websites that are just riddled with holes. Seems like a perfect role and potentially more lucrative than IT (although this is just pure guessing, I haven't compared numbers).
I have explored security world too. Lots of experiments with reversing tools like IDAPro, ollydbg.
Recently a client asked if I can help reverse their own android app because the developers were holding their source code hostage. It was lots of fun and lots of struggle but overall satisfying.
Maybe something like basic security freelance work especially basic webapp security(xss,sqli,csrf, the likes) and some process security with maintaining proper logins, maintaining and rotating credentials etc.
Thank you for your ideas.
Recently a client asked if I can help reverse their own android app because the developers were holding their source code hostage. It was lots of fun and lots of struggle but overall satisfying.
Maybe something like basic security freelance work especially basic webapp security(xss,sqli,csrf, the likes) and some process security with maintaining proper logins, maintaining and rotating credentials etc.
Thank you for your ideas.
Thank you for your insights. The firefighters analogy is definitely very refreshing take on a role which I didn’t consider much before. I’ll definitely explore opportunities on this front.
Maybe you’d enjoy software consulting work under a firm with good engineering management. You get to touch many different tools and tech depending on project.
the mythical 'full stack' engineer? or just sell the AL/ML snake oil and get hired to brute force tech debt.
There’s nothing mythical about it. Just have been doing this for a long time across different parts of the stack and few indie products under the belt. The newer javascript frameworks are a bit tricky to wrap your head around though.
your head is meant to fall into them like a box.... not wrap around it like a present, and in that understanding there is a solace in knowing that javascript is not "the way"
This is the way of beginners. The truth is, read the source code of these libraries. You'll realize a lot of stuff on SO is just copypasta work-arounds, while a more elegant, more efficient solution may exist that just wasn't documented well for your use case.
Perhaps a job at a medium size non-tech company. There are plenty of problems that can be solved with small scale (web) apps everywhere, that don't need webscale levels of scalability, but are well suited for a single full stack developer
Easiest path for you:
Frontend/full stack agency developer (node/react).
Frontend/full stack agency developer (node/react).
This basically outlines my current job. Huge system with many, constantly moving/upgrading parts and services across teams, all while utilizing dozens of internal services, tools, and navigating corporate policies. I’m thriving in it, but totally recognize it’s a steep learning curve and takes longer to onboard newcomers.
Eventually you get to enjoy deprecating old services as much as building new ones, simply because you never have to teach others about them again.
Eventually you get to enjoy deprecating old services as much as building new ones, simply because you never have to teach others about them again.
> Steveslist currently has a very simple and slightly fragile cron setup. We have a single “scheduled jobs server”. We use crontab on this server ... This setup isn’t scaling very well ... We’re considering setting up a new system using a modern tool like Kubernetes.
Do people use Kubernetes for running scheduled jobs like this? It seems like it'd be overkill but in saying that I'm not sure if I know of anything that can be used for running scheduled tasks in a reliable and observable way that's scalable. Maybe Jenkins?
Do people use Kubernetes for running scheduled jobs like this? It seems like it'd be overkill but in saying that I'm not sure if I know of anything that can be used for running scheduled tasks in a reliable and observable way that's scalable. Maybe Jenkins?
this is a pretty common use-case for ci/cd tools.
azure devops, jenkins, concourse, drone, gitlab, ...
also a lot of etl tooling has this stuff (airflow comes to mind immediately)
there's also nomad which is easy to run and can schedule a lot of jobs pretty quickly, with decent observability.
there's... a surprising amount of tooling in this exact space!
azure devops, jenkins, concourse, drone, gitlab, ...
also a lot of etl tooling has this stuff (airflow comes to mind immediately)
there's also nomad which is easy to run and can schedule a lot of jobs pretty quickly, with decent observability.
there's... a surprising amount of tooling in this exact space!
The biggest Kubernetes cluster (arguably) is basically one huge batch job - https://cloud.google.com/blog/products/containers-kubernetes...
> Do people use Kubernetes for running scheduled jobs like this?
We do, mostly because we already have Kubernetes and automation to deploy to it, so why not? The reliablity and monitoring are better than something we'd cook up on our own.
We do, mostly because we already have Kubernetes and automation to deploy to it, so why not? The reliablity and monitoring are better than something we'd cook up on our own.
For better or worse, sure. Here's one case study- https://stripe.com/blog/operating-kubernetes
Kubernetes include cronjobs as an object type. Meaning, this functionality is built in. If you're already running Kubernetes, it's a very simple way to do schedule jobs without needing new management tech.
How can I gain practical experience in these things? The jobs I've had mainly revolve around adding new features, not doing any of the things described in the article. Does working at bigger companies actually give you experience with this?
Enterprise architects usually do not work at large enterprises, but consult to them. In consulting you typically would have risen through the dev stages (junior 1st year, medior 2nd-3rd, senior from 2nd or 3rd onwards (ikr), then onto solution architect (tech lead on small projects ~300k), a few technical analyst team roles on large projects (several M$), experience in a key role in such a project such as Lead Integration Architect, en then onto Enterprise Architect jobs. Note that for anything beyond the solution architect stage your political and managerial skills will become as or even more important than your technical progression.
Alternatively you could be the tech person in a 2 man startup that lucks onto success and grab onto your seat in a wild ride and just never let go.
Alternatively you could be the tech person in a 2 man startup that lucks onto success and grab onto your seat in a wild ride and just never let go.
Where I work, we have Architects but they often play a supporting/consulting role and the teams are expected to learn and implement most of the things listed in this article. They will push you to make good decisions, and will want to vet your designs, but they aren't dictating this stuff from on high (unless of course you're making really poor choices). It's been great because no matter who you are or what role you play on the team (I'm primarily a FE dev) you can learn as much of this stuff as you want. I have to imagine its the same at other companies, so anyway, what I'm saying is you can get a ton of experience in all of this kind of stuff without being an actual Architect at some places (we're a large company). The flip side of that coin here is that it's VERY difficult to get the actual title of Architect; it's a very limited role and it takes a lot more than just systems knowledge to get there (i.e. politics). A lot of the people I've worked with who are Senior level could probably easily transition into a systems architect position at other companies. I'm not sure if this is the best way to get there if that's what you're interested in though, just giving you some food for thought. I don't know how you identify companies where you can get this kind of experience, but if you're interviewing you should ask about "you build it you run it" or stuff like that to see how they manage their infrastructure, that might give you a hint.
Build something that exercises the gaps you want to fill and you'll learn from it. Make your code open source and there's your proof of knowledge. Unless the thing you build is in production with reasonable traffic you won't get to truly experience some of the good or not-so-good decisions you made but at least you made decisions and have some experience.
Yes and no, depends. It gives such experience to a software architect, otherwise not really. Tho, software architects do a lot more than just systems design (tho, that is pretty much their main task). But yes, you get a job as a software architect or something like an "in-training" one, helping one, etc. at a big corporation to get such experience (I'd argue that small companies only need some code design, you want a lot of systems/software design you should go for a big corporation)
You evolve into the architect role as an IC. No new grad is hired like this. You design things and keep going up in scale.
It happens from time to time, but yes - there is no specific job / hiring for this.
The default/common path is evolving into the architect role which seems like a "bad" process of developing an architect.
The best way to develop a new architect, is to have him learn alongside a mentor/teacher who is an architect himself. Developing and architecture are very different jobs which require different mindsets and skills. Also, I've seen many places, teams, etc where people are mostly made to implement feature after feature with no time in between for learning, self-development, courses, etc.
Otherwise, after you've just "arrived" to that architect role you start learning on your own what architecture really is and means.
The default/common path is evolving into the architect role which seems like a "bad" process of developing an architect.
The best way to develop a new architect, is to have him learn alongside a mentor/teacher who is an architect himself. Developing and architecture are very different jobs which require different mindsets and skills. Also, I've seen many places, teams, etc where people are mostly made to implement feature after feature with no time in between for learning, self-development, courses, etc.
Otherwise, after you've just "arrived" to that architect role you start learning on your own what architecture really is and means.
Developing and architecture are absolutely related. Many ideas seem great on paper till you go and implement them and find out the pain points. All the great architects have often been in the weeds of their systems.
> Does working at bigger companies actually give you experience with this?
If you can find someone who needs something for their business or organization, you'll learn a lot of this by working backwards from what they need and building it. So I would suggest the opposite: the smaller the organization the more you can jump around and set up the basics of all these things.
If you can find someone who needs something for their business or organization, you'll learn a lot of this by working backwards from what they need and building it. So I would suggest the opposite: the smaller the organization the more you can jump around and set up the basics of all these things.
Work at a startup as the systems person, because they don't already have one
Regarding DB backups, it would be useful to add that it is not enough to simply make backups, one should also test that the backups are actually restorable.
also the restore/recover process has to be laid out step by step and be as simple as possible. No one is doing a restore in a relaxed, easy going state of mind. Restores are done in full panic mode with the team thinking about how they're going to break the news to their families that they've lost their job.
real developers never concern themselves with the realities of operations /s
Just recently started messing with AWS Amplify. It's crazy how easy it makes basically all this stuff. It really makes it simple to be a solo founder and manage every piece of the stack.
I love the phrase "Advanced Beginner"
Don’t forget a system for organizing contributions to all this code, a system to ensure compliance with the many legal regimes “Steveslist” is offered in, a system for onboarding new hires, a system for expanding market share, a system for keeping money and reporting on it to wouldbe investors, and a system of systems designers that know how to respond to changing and ever expanding requirements.
Don’t sell yourself short: there are systems everywhere you look, not just where there’s convenient precedent to make a web service.
Don’t sell yourself short: there are systems everywhere you look, not just where there’s convenient precedent to make a web service.
This is the best thing I have read in 2020. As a bug bounty Hunter this is insanely helpful to me .
> To do this, we need to write database queries that aggregate over the entirety of our data. We don’t want to run these queries against our production SQL database, because they could put an enormous amount of load on it. We don’t want a huge query issued by an internal analyst to be able to bring our production database to a grinding halt
What kind of query would you have to write to bring down a production db? What makes a solution like hive much better - I guess its optimized for this?
What kind of query would you have to write to bring down a production db? What makes a solution like hive much better - I guess its optimized for this?
> What kind of query would you have to write to bring down a production db?
Scans and Sorts, seen in a query plan, are relatively expensive to run in a production row store. OLAP queries (GROUP BY with aggregate functions like COUNT, SUM, and AVG) do large^/full table scans by definition. They take seconds to run while your goal in a Cloud OLTP system is thousands of requests per second. An automatic sort issued per query in an OLTP system is pathological and represents a vector for a DoS attack.
> What makes a solution like hive much better - I guess its optimized for this?
Column stores use compressed bitmap indexes that are optimized for scans over a small number of columns. Hive is SQL over Hadoop, and is inherently slow but it does offload the processing from your Production OLTP server. Hive supports the RCFile format which is partially column oriented. The ORC file format is fully column oriented, replaces RCFile format, but requires Presto (or equivalent). Hive is brownfield for existing Hadoop clusters but it has no place in a discussion about greenfield architecture other than discussing historical systems.
If you have a need for GROUP BY style analytics, a true column store like Presto, Impala, or RedShift is a necessity.
^EDIT: based on zbentley's comment
Scans and Sorts, seen in a query plan, are relatively expensive to run in a production row store. OLAP queries (GROUP BY with aggregate functions like COUNT, SUM, and AVG) do large^/full table scans by definition. They take seconds to run while your goal in a Cloud OLTP system is thousands of requests per second. An automatic sort issued per query in an OLTP system is pathological and represents a vector for a DoS attack.
> What makes a solution like hive much better - I guess its optimized for this?
Column stores use compressed bitmap indexes that are optimized for scans over a small number of columns. Hive is SQL over Hadoop, and is inherently slow but it does offload the processing from your Production OLTP server. Hive supports the RCFile format which is partially column oriented. The ORC file format is fully column oriented, replaces RCFile format, but requires Presto (or equivalent). Hive is brownfield for existing Hadoop clusters but it has no place in a discussion about greenfield architecture other than discussing historical systems.
If you have a need for GROUP BY style analytics, a true column store like Presto, Impala, or RedShift is a necessity.
^EDIT: based on zbentley's comment
> OLAP queries (GROUP BY with aggregate functions like COUNT, SUM, and AVG) do full table scans by definition
Isn't it only a full table scan if your query isn't otherwise filtered? Those functions have to read every row of "something", but that something might not always be a whole table.
Isn't it only a full table scan if your query isn't otherwise filtered? Those functions have to read every row of "something", but that something might not always be a whole table.
We started evaluating clickhouse, after saying that sentry it's using it
Systems Design is a formidable topic because it can go in so many directions. This is a good guide. Might I also recommend something that includes caching, partitioning, indexing, and NoSQL vs SQL. More detail can be found here https://www.educative.io/courses/grokking-the-system-design-...
I remember using the course you have linked to some time ago - it was very useful. I noticed now that the course is sold on a subscription model. I don't remember being that the case. Would you happen to know when that changed?
I don't. I came across this in 2018 and recall the course being for sale for $80.
> We’re considering either splitting up our cron jobs into multiple servers, or setting up a new system using a modern tool like Kubernetes.
:facepalm: beanstalkd can handle a pretty massive amount of jobs before you need to start worrying about scale. I love the huge jump from simple to complex.
:facepalm: beanstalkd can handle a pretty massive amount of jobs before you need to start worrying about scale. I love the huge jump from simple to complex.
To be fair they offered a simple option (splitting crontabs) as well.
Really like the prose/Adventure-like writing style of this "tutorial"!
> database engines that are quick at small queries are typically unacceptably slow of answering giant queries
Even with proper indexing? I haven't seen this issue with Postgres but maybe I wasn't working on large enough data sets.
Even with proper indexing? I haven't seen this issue with Postgres but maybe I wasn't working on large enough data sets.
off topic: I like the simplistic design of this website. Can someone please help me understand if this is hosted on WordPress or something like that?
It might be Jekyll according to https://robertheaton.com/2014/07/26/lessons-from-a-surprisin...
Looking at the page source, it doesn't appear to be Wordpress; none of the usual cruft that WP scatters through the source (though it could have been processed via a filter).
Looking at the request/response headers in Firefox's web developer tools, it's gone through Cloudflare, but there's also a x-github header, so maybe some custom tooling that's pushed content to github?
Looking at the request/response headers in Firefox's web developer tools, it's gone through Cloudflare, but there's also a x-github header, so maybe some custom tooling that's pushed content to github?
It's jekyll: https://github.com/robert/robert.github.com
Nowadays, many sites are built using a static site generator. Have a look at Hugo, Jekyll, Gatsby, etc.
Usually you have plenty of themes to choose from. I use Hugo and can tell you there are many themes similar to the one in this website.
Usually you have plenty of themes to choose from. I use Hugo and can tell you there are many themes similar to the one in this website.
This is amazing. Thank you Robert.
Great post! Thanks for sharing.
Great post!
anyone else just getting a blank page here?
no DDD?
[deleted]
1. Prototype and benchmark each of your stack pieces before you pick a stack. It is far easier to fix architecture mistakes when you don’t have 10000 users expecting overnight customer service. If you are using new technology, find good open source products to see how they’ve structured their projects. Your architecture, designed for speed and experience, will be your key differentiator. Inherent speed at core task, design to user needs, and name choice are the 3 musketeers of a solid growth.
2. Prepare for abusers to attack your system from every direction, especially if you enable users to publish content under your domain. You will see bots looking for Wordpress installations, users trying to fill content with SEO links, users trying every hacking vector known to market. Collect known vectors and test for them and never refuse a legitimate bug bounty request.
3. There is an eternal debate about the trade off between building a quality product at start or opening early to get user feedback before you get too deep into features. There is merit to both choices as a solo founder. The moment you open the gates to users, your ability to make changes comes with very high friction. With time, trying new features becomes a tremendous luxury hidden under bug requests, roadmap, customer service replies, etc. Build your biggest riskiest assumptions first.
4. Testing is hard as a solo founder. Selenium is your friend. If you don’t spend time with them, your users will take that time in multiples after a mistake.
The best way to learn is to come up with a product you really want and build it in your own time. You can test launch in a weekend.
When I started launching consumer sites solo as an engineer, I went from a tight specialization to being unafraid to try any tech if it gives me an advantage in solving a problem. Once you’ve simulated enough problems and dealt with the consequences of your choices personally, you can play that 10 level chess game with the architecture of each new feature much faster.