Ask HN: Where to learn Big Tech engineering, without working at Big Tech?
11 comments
Are you sure the big tech way is the best for your start up? Different sized businesses have very different needs.
It's not really about scalability on the TPS side for us, it's more of the scalability of the engineering practice, maintainability and developer productivity.
Big Tech gets a lot of things right.
Big Tech gets a lot of things right.
If you can afford it, I strongly recommend Bradfield School of Computer Science https://bradfieldcs.com/
[deleted]
What kind of company r u now building ?
Stop thinking that Big Tech engineering works for everyone
I have not worked with FAANG or any project that requires massive web scale. Most of my career I have worked with resource constraint devices. So when I decided to write a web app, it was a horror show for me. Even though I have a CS background and understand any tech when I read and see it, it becomes extremely difficult to navigate the maze of different options available and the many things one is forced to consider to get a web app running.
What I have ultimately realised is that having a middle path to software architecture development is the optimal way i.e. build critical sections on your own and outsource others. For example, at the start do not think about horizontal scalability, it is a nightmare scenario to handle even if outsourced. So concentrate of vertical scalability with the help of a hosting provider. Horizontal scalability is a luxury that you can afford when you have dedicated people to manage it. There is a reason people developing Postgres have resisted the temptation to support horizontal scalability. It is a very hard problem to solve.
So if you are starting small and at an early stage then concentrate on how you will be able to own the data, how you will be able to own the business logic, and how you will be able to migrate from one service provider to another avoiding any vendor lock-in.
High availability, fault tolerance, always-on architectures are very difficult to achieve and maintain. There is a reason why FAANG spend so much resources on this and still get it wrong.
In case you want to go that route, then for some practical advise refer to https://sre.google
After evaluating lots of HA and fault-tolerant solutions and thinking about it for some time, personally I have come to the conclusion that having a single node with postgres, backed by streaming replication for backup, with dynamic vertical scaling using a hosting service where I can easily migrate my setup to other hosting provider is the way to go.
If your journey has made you to come to some other conclusion then please do share.
What I have ultimately realised is that having a middle path to software architecture development is the optimal way i.e. build critical sections on your own and outsource others. For example, at the start do not think about horizontal scalability, it is a nightmare scenario to handle even if outsourced. So concentrate of vertical scalability with the help of a hosting provider. Horizontal scalability is a luxury that you can afford when you have dedicated people to manage it. There is a reason people developing Postgres have resisted the temptation to support horizontal scalability. It is a very hard problem to solve.
So if you are starting small and at an early stage then concentrate on how you will be able to own the data, how you will be able to own the business logic, and how you will be able to migrate from one service provider to another avoiding any vendor lock-in.
High availability, fault tolerance, always-on architectures are very difficult to achieve and maintain. There is a reason why FAANG spend so much resources on this and still get it wrong.
In case you want to go that route, then for some practical advise refer to https://sre.google
After evaluating lots of HA and fault-tolerant solutions and thinking about it for some time, personally I have come to the conclusion that having a single node with postgres, backed by streaming replication for backup, with dynamic vertical scaling using a hosting service where I can easily migrate my setup to other hosting provider is the way to go.
If your journey has made you to come to some other conclusion then please do share.
>High availability, fault tolerance, always-on architectures are very difficult to achieve and maintain.
Ironically, the only reason this is true is because generally people working on those pretty much cobble shit together instead of writing correct code, so you have subsystems of code talking to subsystem of code, and inevitably shit goes wrong.
Build your system correctly to the use case, from the ground up, and you will have a much easier time.
Of course the counter argument to this is the age old "what if developers make mistakes, its better to use off the shelf software thats already tested and verified". But the real answer to that isn't to use off the shelf software, its to not be or hire shit developers.
Ironically, the only reason this is true is because generally people working on those pretty much cobble shit together instead of writing correct code, so you have subsystems of code talking to subsystem of code, and inevitably shit goes wrong.
Build your system correctly to the use case, from the ground up, and you will have a much easier time.
Of course the counter argument to this is the age old "what if developers make mistakes, its better to use off the shelf software thats already tested and verified". But the real answer to that isn't to use off the shelf software, its to not be or hire shit developers.
I think that the answer is, "you can't, actually."
(I worked at a MAAMA for 3 years).
The problem, as I kept pointing out to anyone who would listen at my BigCorp, was that about 80% of what was necessary to know was what one colleague called 'tribal knowledge'.
This is just a weird way of saying, "we couldn't be bothered to write it down, you have to know a guy."
Incidentally, aside from being inefficient and bus-factor dangerous, it's also murder for any "diverse hire", because, you guessed it, you're not part of the tribe.
(I worked at a MAAMA for 3 years).
The problem, as I kept pointing out to anyone who would listen at my BigCorp, was that about 80% of what was necessary to know was what one colleague called 'tribal knowledge'.
This is just a weird way of saying, "we couldn't be bothered to write it down, you have to know a guy."
Incidentally, aside from being inefficient and bus-factor dangerous, it's also murder for any "diverse hire", because, you guessed it, you're not part of the tribe.
This. There is certainly a Catch-22 here - to get hired by a Tech Giant, one ideally needs the knowledge of how to solve problems at Tech Giant scale, but that really only comes after being hired at a Tech Giant that has the kind of money and 'tribal knowledge' that's needed to teach one how to solve problems at Tech Giant scale (sorry, but your average community college or business really can't provide that!). On a much smaller scale, the same can be said about many professions, including the trades, but an electrical contractor will be less persnickety about giving a chance to an 'apprentice' than a Tech Giant that's really only interested in hiring employees who they think can on average contribute $1+ million in revenue to the company. If you still want to work for Google, Microsoft, Facebook, Apple, Amazon though, you can always apply to be a contract worker thru one of their vendors (coder/tester, security guard, delivery person, food service worker, etc.).
I'm not affiliated with him in any way, but I quite like the tweets from http://twitter.com/GergelyOrosz
He has a book that I've not read, but could be worth getting https://blog.pragmaticengineer.com/
FWIW I've worked at FAANG for 5+ years.
He has a book that I've not read, but could be worth getting https://blog.pragmaticengineer.com/
FWIW I've worked at FAANG for 5+ years.
But where do we learn how things are done in the trenches in Big Tech companies like FAANGMULA?
Sure, there are eng blogs, but those are such a small subset of resources and too high level, if you search for anything practical (e.g. trying to make an architectural decision), they almost never show up in search results. They're mostly a hiring tool rather than really trying to help anyone.
Sure there are books like DDIA, but they're very academic and definitely not searchable when you have practical questions.
I feel like whenever I search for tech stuff on Google, the majority of the stuff on the first page are content-milled articles from India.
For context, I've worked at FAANGMULA for many years, and am now building a tech company. I know in general how things are done technically and have a taste of what good solutions look like, but some of the stuff that are further removed from my day-to-day I need to either refresh on or learn about, but high quality resources like this are pretty hard to locate online.