Web Architecture 101(engineering.videoblocks.com)
engineering.videoblocks.com
Web Architecture 101
https://engineering.videoblocks.com/web-architecture-101-a3224e126947/?ref=abhimanyu
48 comments
Amen, web content to learn from wise security, back end stuff, architecture doesn't get enough coverage.
There are so many super simple app tutorials that are an island to themselves that might get your other super simple app going, but they don't even give lip service to any deeper concepts or what is going on the back end.
I have a theory that the volume of basic copy this code type tutorials is because they are really resume fodder.... I've run through some tutorials that are downright misleading IMO, and I could tell that as a noob myself.
I was playing with one recently that I wondered, hey can I make it do X too? But the code was laid out and ran so wonky that it was just easier for me to start fresh on my own. It was a react tutorial and even running it as provided it ran but react was all wtfareyodoing.jpg in the console... right out of the tutorial... come on.
I wish people would quit it with the code tutorials as resume content.
As for security I finished a boot camp recently and I asked about security, the answer "We really don't address security."
I have a networking background so I could see a lot that was missing that folks really should know...but not even given lip service.
There are so many super simple app tutorials that are an island to themselves that might get your other super simple app going, but they don't even give lip service to any deeper concepts or what is going on the back end.
I have a theory that the volume of basic copy this code type tutorials is because they are really resume fodder.... I've run through some tutorials that are downright misleading IMO, and I could tell that as a noob myself.
I was playing with one recently that I wondered, hey can I make it do X too? But the code was laid out and ran so wonky that it was just easier for me to start fresh on my own. It was a react tutorial and even running it as provided it ran but react was all wtfareyodoing.jpg in the console... right out of the tutorial... come on.
I wish people would quit it with the code tutorials as resume content.
As for security I finished a boot camp recently and I asked about security, the answer "We really don't address security."
I have a networking background so I could see a lot that was missing that folks really should know...but not even given lip service.
I'm guessing, but IMO when you're dealing with load balancing, services and data pipelines you're at a scale where you should be able to hire qualified people to do the job so the docs are geared towards that audience.
You're probably right re load balancing, data pipelines and some services, but i think security is important regardless of scale, esp if it's an enterprise app.
Even simpler concepts like structuring and deployment of larger apps, database migration, creating dev, staging and production environments, test development etc would be really useful. These things are important even at low scale, and generally absent from tutorials. Learning this on your own without mentors or CS background creates lots of issues
Even simpler concepts like structuring and deployment of larger apps, database migration, creating dev, staging and production environments, test development etc would be really useful. These things are important even at low scale, and generally absent from tutorials. Learning this on your own without mentors or CS background creates lots of issues
I've seen comments on this article elsewhere around load balancing not being needed until you're at scale, but do others not put it in quickly, not for volume, but for availability?
Again - when you need something to be available you probably want someone who knows what he's doing from start setting it up.
Not saying you can't or shouldn't do it but it's not surprising that there's not a lot of newbie friendly tutorials for such things.
Not saying you can't or shouldn't do it but it's not surprising that there's not a lot of newbie friendly tutorials for such things.
Setting up HA LBs isn't hard though. Two load balanced IPS in your DNS hitting each external load balancer IP, LBs have keepalived configured with two floating VIPs each. I learnt that on the fly when having to build an HA service without the economic resources usually associated with doing so. ( Databases and routers/firewalls were a whole different beast though)
I too am in great need of things like that.
I'm a mobile dev developing an app that I think I'm going to quit my job for to work on full time. I'm 33 and in over a decade of doing mobile (and front-end web) programming I've never worked on a personal project that showed as much promise as this one has for me, both financially and from a fulfillment standpoint.
However the prospect of wrangling the back-end and all the new things I need to do around deploying, scaling, containers, separation of services: cache, API, queue, DB, and making it all secure on top of that is giving me pause and making me less confident the more I read :(
I'm a mobile dev developing an app that I think I'm going to quit my job for to work on full time. I'm 33 and in over a decade of doing mobile (and front-end web) programming I've never worked on a personal project that showed as much promise as this one has for me, both financially and from a fulfillment standpoint.
However the prospect of wrangling the back-end and all the new things I need to do around deploying, scaling, containers, separation of services: cache, API, queue, DB, and making it all secure on top of that is giving me pause and making me less confident the more I read :(
Nah, don't let that discourage you. You can probably start with a Postgres backend that will serve as your db, full text search, data warehouse, and even job queue. Load balancers in front of your stack are available from aws, etc. and are easy to set up. The rest, like a caching redis server, actual job queue, etc. you can add as you go if it becomes needed. Some language stacks let you run async operations and queues in the server itself which may be enough. There is no need for containers or complicated deployment processes. If you're on the JVM, for example, just create an uberjar and run that. Focus on things that actually bring value to your business and use the simplest, most minimal solution everywhere else. Don't give up on your idea because you feel overwhelmed. Good luck.
You've described the exact approach I've used for my B2B SaaS.
Initially we had a web server (two actually, to achieve zero-downtime deploys) and a database.
A few months later, we introduced Redis for caching. This was much earlier than I expected we'd need to - these days as soon as a web app gets traction, it starts attracting scammers, spammers, and manipulators who put unexpected load on your system.
Just this week, we've finally added a worker server to run long running tasks. Until this week, Java's brilliant Executors were handling this for us in a background thread.
A Java uberjar, deployed to AWS's Elastic Beanstalk, is an approach I'd highly recommend for anyone wanting to get version 1 of their web app deployed.
Initially we had a web server (two actually, to achieve zero-downtime deploys) and a database.
A few months later, we introduced Redis for caching. This was much earlier than I expected we'd need to - these days as soon as a web app gets traction, it starts attracting scammers, spammers, and manipulators who put unexpected load on your system.
Just this week, we've finally added a worker server to run long running tasks. Until this week, Java's brilliant Executors were handling this for us in a background thread.
A Java uberjar, deployed to AWS's Elastic Beanstalk, is an approach I'd highly recommend for anyone wanting to get version 1 of their web app deployed.
Was about to write this exact reply. A single database and web server goes surprisingly far.
If you have a reasonable idea of what you are doing.
Even if you don't, throwing some more hardware at it can still get you pretty far with a decent ROI in the early stages.
thanks for the advice - yes I'm on the JVM w/ spring boot.
I suppose I may be overcomplicating things with trying to do it "the right way" right out of the gate. I guess the "perfect is the enemy of done" saying comes into play here.
I suppose I may be overcomplicating things with trying to do it "the right way" right out of the gate. I guess the "perfect is the enemy of done" saying comes into play here.
Experiment with abstracting the shit out of RabbitMQ to solve any business logic problem. I'm in a different CS field than web development but whenever I've had to implement something business related it almost always was just highly abstract message brokering with React hacked together UI. I learned UI from reading lecture notes on 'Human Computer Interface' design from the plenty of university courses around offering this subject http://www.cs.cmu.edu/~bam/uicourse/05863fall17/schedule.htm...
For learning web app APIs all I did was search Google Scholar for 'API design' and read whatever was available. Joshua Bloch has a good paper on API design. As for scaling, containers, concurrency ect look at Elixir w/Phoenix framework instead, or Erlang w/Yaws and just run it on a cheap FreeBSD comanaged box or deploy Phoenix to Heroku. BEAM will take care of the scaling. There's books around https://pragprog.com/book/lhelph/functional-web-development-...
For learning web app APIs all I did was search Google Scholar for 'API design' and read whatever was available. Joshua Bloch has a good paper on API design. As for scaling, containers, concurrency ect look at Elixir w/Phoenix framework instead, or Erlang w/Yaws and just run it on a cheap FreeBSD comanaged box or deploy Phoenix to Heroku. BEAM will take care of the scaling. There's books around https://pragprog.com/book/lhelph/functional-web-development-...
I found the book "Designing Data-Intensive Applications" by Martin Kleppmann to be a good, practical primer on a lot of these topics.
If you want to go deeper on any of the subjects he discusses, his references for every chapter are solid and provide a deeper understanding.
If you want to go deeper on any of the subjects he discusses, his references for every chapter are solid and provide a deeper understanding.
I'm a big fan of DDIA (I have a copy in front of me right now), but the topics covered in DDIA are much more around distributed systems and databases, in short "data architecture", i.e. batch, streaming, derived data, transactions, consistency, etc.
The article above is more about "web architecture".
There is some overlap but DDIA largely doesn't really cover the same topics.
The article above is more about "web architecture".
There is some overlap but DDIA largely doesn't really cover the same topics.
Any other books you'd recommend that come closer to fully covering the topic this post discusses?
Perhaps "Web Scalability for Startup Engineers"
by Artur Ejsmont. But I admit I have't read it yet -- I however felt the same as you and started exploring for more resources after coming across this thread.
Designing for Scalability with Erlang/OTP book is good resource. I mix Erlang with Algorithmia pay-per-use algorithms these days to drop in features like sales analysis.
A modern version of Cal Henderson's "Building Scalable Web Sites" would be nice to have. I'm sure a lot of it still applies, but it was written a decade ago: http://shop.oreilly.com/product/9780596102357.do
Not really, but I don't know this space as well as I do the space covered by DDIA. It could well be that no one has written such a book, and perhaps the market is ready for one.
In fact, the Unix and Linux System Administration Handbook offers a pretty good coverage of this topic.
Sure, it's a great book, but it only covers a few of these topics (maybe 4, 6a, 5, and 9) and covers a lot that goes beyond them. It's a book I would heartily recommend, but not for someone whose primary goal was to learn about web architecture.
Any other books you'd recommend that come closer to fully covering the topic this post discusses?
Clean Architecture
https://www.goodreads.com/book/show/18043011-clean-architect...
The Art of Scalability (2nd edition) https://www.goodreads.com/book/show/25686275-the-art-of-scal...
The Art of Scalability (2nd edition) https://www.goodreads.com/book/show/25686275-the-art-of-scal...
For anyone interested in building or modifying system architectures I also highly recommend this book, although it isn't web-specific (similar to the DDIA recommendation):
Building Evolutionary Architectures https://www.goodreads.com/book/show/36308520-building-evolut...
Building Evolutionary Architectures https://www.goodreads.com/book/show/36308520-building-evolut...
Good question! The Practice of Cloud System Administration is one that comes to mind, but it's not precisely focused on this topic either.
https://the-cloud-book.com/
Edit: thinkmassive mentioned the Art of Scalability, which I'd also recommend.
https://the-cloud-book.com/
Edit: thinkmassive mentioned the Art of Scalability, which I'd also recommend.
Can anyone recommend an introduction to web architecture in the more general and less complex sense? I would like to learn about how the browser interacts with the server (I don't care about load balancing or caching for now), how JS interacts with HTML and CSS, how API's work?
I know a common architecture is JS in the browser making requests to the server and receives JSON and then renders it somehow. Sometime the server renders stuff itself and sends it to the browser, right?
And then there's possibly a database attached to the server.
How does all that work?
I know a common architecture is JS in the browser making requests to the server and receives JSON and then renders it somehow. Sometime the server renders stuff itself and sends it to the browser, right?
And then there's possibly a database attached to the server.
How does all that work?
I wrote an article on learning web development, might be useful to you. Here's the link: https://medium.com/career-change-coder/how-i-broke-into-web-...
If you're only looking for high-level concepts, Crash Course has some good episodes about the workings of the internet.
If you're only looking for high-level concepts, Crash Course has some good episodes about the workings of the internet.
Thank you, some of the references were helpful!
I'd love to see the initial diagram grown step by step. You don't start out this way and it can be extremely informative to see which bits get added when.
It depends on the business needs. Generally what happens is a developer will notice that something is going to become a problem and start considering solutions, long before the app becomes unmaintainable. It's all driven by what the application needs.
Random thought, but when you see that.. it also makes more visible a little bit more, what it means to be responsible for the "full-stack" in 2018.
Skyscanners "Full Stack Test" was entirely in Javascript when I got asked to do it. Screw that, its not my strong point. Apparently the alternative is Hackerrank for systems engineers when I mentioned that to them. Hiring in our industry is a shambles.
I'm sure this has been said before but this reminds me of https://www.youtube.com/watch?v=CJQU22Ttpwc
This is a great overview! It's a shame this sort of thing isn't taught in schools (from my very own anecdotal university experience).
While I agree that a precursor to these concepts would be a great initiator, however I also believe that a good appreciation for scaling technologies can only be developed by getting hands on and working with a high traffic website. Just reading the theory might not help that well.
These things were taught at my university (Information Systems major), but the info was pretty out-of-date and not as practical as this article.
[deleted]
"Web Application Architecture" by Leon Shklar is a good read for a detailed description.
Really enjoyed it... Any recommendations for Web architecture 102 ?
Without committing to an entire book, perhaps more detail on load-balancing and proxies?
https://blog.envoyproxy.io/introduction-to-modern-network-lo...
https://blog.envoyproxy.io/introduction-to-modern-network-lo...
I posted this above in reply to someone else asking the same question:
> Perhaps "Web Scalability for Startup Engineers" by Artur Ejsmont. But I admit I have't read it yet -- I however felt the same as you and started exploring for more resources after coming across this thread.
> Perhaps "Web Scalability for Startup Engineers" by Artur Ejsmont. But I admit I have't read it yet -- I however felt the same as you and started exploring for more resources after coming across this thread.
This is overcomplicated, you can do it much simpler using some Backend as a Service, like Parse, Hoodie, Backendless or Firebase with a client-side business logic.
This is great. It's great for new devs - and even for upper management who may want a simple/executive summary of how modern stacks work.
Funny.. I just had to diagram the same kind of generic thing at my day job. And it's almost exactly the same.
Nicely done and a good share!
Nicely done and a good share!
Best job in the world i can't believe they pay me to do this!
For people who are working in non-tech jobs but teaching themselves programming so they can build useful tools for themselves, colleagues and customers, I think there's a big gap in learning tools. Often these ppl can have enterprise users out of the box bc they understand needs and know people, but can only make hackish solutions that are useful but don't scale well or aren't sufficiently secure. These ppl don't have CS backgrounds and don't code for a living, so going straight into docs can be really overwhelming even if it's the "right" thing to do, esp if it's an architectural choice or security issue where the documentation may not be as straightforward