We Don’t Use Docker(launchyourapp.meezeeworkouts.com)
launchyourapp.meezeeworkouts.com
We Don’t Use Docker
https://launchyourapp.meezeeworkouts.com/2021/03/why-we-dont-use-docker-we-dont-need-it.html?m=1
550 comments
Several years ago, I was chatting with another engineer from a close competitor. He told me about how they'd set up a system to run hundreds of data processing jobs a day over a dozen machines, using docker, load balancing, a bunch of AWS stuff. I knew these jobs very well, they were basically identical for any company in the space.
He then mentioned that he'd noticed that somehow my employer had been processing thousands of jobs, much faster than his, and asked how many machines we were using.
I didn't have the heart to tell him we were running everything manually on my two-year-old macbook air.
He then mentioned that he'd noticed that somehow my employer had been processing thousands of jobs, much faster than his, and asked how many machines we were using.
I didn't have the heart to tell him we were running everything manually on my two-year-old macbook air.
F- me. I love this. This is a really important message.
It's like Jonathon Blow asks: why does Photoshop take longer to load today than it did in the 90s despite the (insane) advances in hardware?
I believe it's due to a bunch of things, but over complicating the entire process is one of the big issues. If people (developers/engineers) would only sit back and realise just how much computing power they have available to them, and then realised that if they kept things simple and efficient, they could build blazing fast solutions over night.
I cringe thinking about the wasted opportunities out there.
It's like Jonathon Blow asks: why does Photoshop take longer to load today than it did in the 90s despite the (insane) advances in hardware?
I believe it's due to a bunch of things, but over complicating the entire process is one of the big issues. If people (developers/engineers) would only sit back and realise just how much computing power they have available to them, and then realised that if they kept things simple and efficient, they could build blazing fast solutions over night.
I cringe thinking about the wasted opportunities out there.
Jonathan Blow's Preventing the Collapse of Civilization [0] is an excellent talk, on how too much abstractions destroy knowledge.
This is a problem with the software industry today. we have forgotten how to do the simple stuff, that works and is robust.
[0]: https://www.youtube.com/watch?v=ZSRHeXYDLko
This is a problem with the software industry today. we have forgotten how to do the simple stuff, that works and is robust.
[0]: https://www.youtube.com/watch?v=ZSRHeXYDLko
That was a really great watch, thanks!
Compare photoshop between the 90s and now. 10x on features. Size of photos grown exponentially as well.
I see the point you're trying to make, however the increase in features (and their complexity) plus the size of the average graphic that a high-end professional has maybe grown by 300-500% since the 90s. In fact I'll tell you what: I'll give you a growth of 10,000% in file sizes and feature complexity since the since the 90s...
... computational power has grown ~%259,900 since the 90s.
The point being made is this: Photoshop does one job and has one focus should), yet it has gotten slower at doing that one job and not faster. Optimising the code AND introducing incredibly hardware to the consumer market should see Photoshop loading in milliseconds, in my opinion.
... computational power has grown ~%259,900 since the 90s.
The point being made is this: Photoshop does one job and has one focus should), yet it has gotten slower at doing that one job and not faster. Optimising the code AND introducing incredibly hardware to the consumer market should see Photoshop loading in milliseconds, in my opinion.
>The point being made is this: Photoshop does one job and has one focus should), yet it has gotten slower at doing that one job and not faster.
Has it though? Without measurements this is just idle talk.
And I've used Photoshop in the 90s and I use it today ocassionally. I remember having 90s sized web pictures (say, 1024x768) and waiting for a filter to be applied for tens of seconds - which I get instantly today with 24MP and more...
And if we're into idle talk I've always found Photoshop faster in large images and projects than competitors, including "lightweight" ones.
It's hella more optimized than them.
In any case, some e.g. image filter application (that takes, e.g. 20 seconds vs 1 minute with them) just calls some optimized C++ code (perhaps with some asm thrown in) that does just that.
The rest of the "bloat" (in the UI, feature count, etc) has absolutely no bearing as to whether a filter or an operation (like blend, crop, etc) runs fast or not. At worst, it makes going around in the UI to select the operations you want slower.
And in many cases the code implementing a basic operation, filter, etc, hasn't even been changed since 2000 or so (if anything, it was optimized further, taken to use the GPU, etc).
Has it though? Without measurements this is just idle talk.
And I've used Photoshop in the 90s and I use it today ocassionally. I remember having 90s sized web pictures (say, 1024x768) and waiting for a filter to be applied for tens of seconds - which I get instantly today with 24MP and more...
And if we're into idle talk I've always found Photoshop faster in large images and projects than competitors, including "lightweight" ones.
It's hella more optimized than them.
In any case, some e.g. image filter application (that takes, e.g. 20 seconds vs 1 minute with them) just calls some optimized C++ code (perhaps with some asm thrown in) that does just that.
The rest of the "bloat" (in the UI, feature count, etc) has absolutely no bearing as to whether a filter or an operation (like blend, crop, etc) runs fast or not. At worst, it makes going around in the UI to select the operations you want slower.
And in many cases the code implementing a basic operation, filter, etc, hasn't even been changed since 2000 or so (if anything, it was optimized further, taken to use the GPU, etc).
I recall my dad requiring overnight sessions to have Photoshop render a particular filter on his Pentium 166MHz. That could easily take upwards of an hour, and a factor 10 more for the final edit. He'd be working on one photo for a week.
To me it feels as though the last decade and a half computational power has not grown vertically. Instead Intel and AMD have grown computational power horizontally (i.e adding more cores). I'm looking at the difference the M1 has had on compute performance as a sign X86 strayed.
It has also grown substantially vertically: single-core speeds keep going up (about 10x from a decade and a half ago), even as core count increases. (and M1 is not substantially faster than the top x86 cores, the remarkable thing is how power efficient it is at those speeds).
This was state of the art 15 years ago.
https://en.wikipedia.org/wiki/Kentsfield_(microprocessor) Modern processors are nowhere near 10x faster single core. Maybe 4x.
If you take into account performance/watt you do get 10x or better.
https://en.wikipedia.org/wiki/Kentsfield_(microprocessor) Modern processors are nowhere near 10x faster single core. Maybe 4x.
If you take into account performance/watt you do get 10x or better.
The 3.4ghz P4 was released 2004 https://en.m.wikipedia.org/wiki/List_of_Intel_Pentium_4_proc...
If the trend had continued we would have 1500Ghz cores by now.
If the trend had continued we would have 1500Ghz cores by now.
Clock speed != single-threaded performance. Clock speeds plateaued a long time ago, single threaded performance is still improving exponentially (by being able to execute multiple instructions in an instruction stream in parallel, as well as execute the same instructions in less clock cycles), though the exponent approximately halved around 2004 (if the trend had continued we would be at about a 100-500x improvement by now).
https://github.com/karlrupp/microprocessor-trend-data
https://github.com/karlrupp/microprocessor-trend-data
Hard to say it's still "exponential"...what do you think the current constant doubling period is now?
Here's the single thread raw data from that repo. If you take into account clock speed increase (which, as you agree, have plateaued) we're looking at maybe a 2x increase in instructions per clock for conventional int (not vectorized) workloads.
Is there even another 2x IPC increase possible? At any time scale?
https://github.com/karlrupp/microprocessor-trend-data/blob/m...
Here's the single thread raw data from that repo. If you take into account clock speed increase (which, as you agree, have plateaued) we're looking at maybe a 2x increase in instructions per clock for conventional int (not vectorized) workloads.
Is there even another 2x IPC increase possible? At any time scale?
https://github.com/karlrupp/microprocessor-trend-data/blob/m...
No one is choosing Photoshop over competitiors because of load time
And somehow Photopea is fast, free and in browser and suffices for 85% of whatever people do in Adobe Photoshop.
Yes, I use Photoshop because, as it happens, I need to do 100% of my job.
Back in the day I was using Intel Celeron 333 MHz and then AMD Duron 800 MHz.
I did not know how to use Winamp playlists because Winamp has been "an instant" app for me, I just clicked on a song and it played within miliseconds. That was my flow of using Winamp for years. This did not change between the Celeron and Duron, the thing was instant on both Celeron and Duron.
Then Winamp 3 came out and I had to use playlists because a song once clicked took good second or two to start playing. Winamp 5 from 2018 still starts slower than my beloved 2.73* did 20 years ago. On Celeron 333 and 5400 RPM HDD with 256 MB of RAM. I think even the good old Winamp 2.x is not as fast as it was on Windows 98/XP.
Something went wrong.
* not sure if it was 2.73, but I think so
Note: I realise Winamp 3 was crappy as hell, but still...
I did not know how to use Winamp playlists because Winamp has been "an instant" app for me, I just clicked on a song and it played within miliseconds. That was my flow of using Winamp for years. This did not change between the Celeron and Duron, the thing was instant on both Celeron and Duron.
Then Winamp 3 came out and I had to use playlists because a song once clicked took good second or two to start playing. Winamp 5 from 2018 still starts slower than my beloved 2.73* did 20 years ago. On Celeron 333 and 5400 RPM HDD with 256 MB of RAM. I think even the good old Winamp 2.x is not as fast as it was on Windows 98/XP.
Something went wrong.
* not sure if it was 2.73, but I think so
Note: I realise Winamp 3 was crappy as hell, but still...
This is why I did choose and stick to coolplayer at the time (before I converted to Linux) : no install, so light, so fast, and it had everything I needed. I loved it when I could find such an elegant a versatile app. I don't need to get "more" every 6-12 months.
I learned that every time you gain something, you also loose something without realizing it, because you take it for granted.
I learned that every time you gain something, you also loose something without realizing it, because you take it for granted.
Winamp 2.73 is still my default player and works Win10 Pro 64x build $LATEST. I will never change. It is 2MB pure gold.
This is sad to read :(
I remember having a 486 my self and everything being so damn snappy. Sigh.
I remember having a 486 my self and everything being so damn snappy. Sigh.
Are you me? Perhaps a little bit later on and a different set of signifiers (foobar2000/xp/celeron 1.7) but the same idea. Things were so much snappier back then than on my previously-SOTA MBPR 2019. Sigh.
I was at a graphic design tradeshow back in the mid 90's and there was a guy there demonstrating this Photoshop alternative called Live Picture or Live Photos or something like that. And he had a somewhat large, at the time, print image on the screen, probably 16mb or so, and was zooming in and out and resizing the window and it was redrawing almost instantly.
This was AMAZING.
Photoshop at the time would take many many seconds to zoom in/out.
One person in the group asked, "Yeah, but how much memory is in that machine?"
The guy hemmed and hawed a bit and finally said "It's got a bit, but not a lot, just 24mb."
"Yeah, well that explains it." Nobody had 24mb of RAM at that time. Our "big" machine had 16mb.
This was AMAZING.
Photoshop at the time would take many many seconds to zoom in/out.
One person in the group asked, "Yeah, but how much memory is in that machine?"
The guy hemmed and hawed a bit and finally said "It's got a bit, but not a lot, just 24mb."
"Yeah, well that explains it." Nobody had 24mb of RAM at that time. Our "big" machine had 16mb.
Live Picture was the first to use what I think are called image proxies, where you can have an arbitrarily large image and only work with the screen-image sized image. Once you have applied all the changes and click save, it will then grind through the full size image if needed.
A feature that Photoshop has since added, but it appeared in Live Picture first.
A feature that Photoshop has since added, but it appeared in Live Picture first.
Yup, that must have been it. I think Adobe implemented that as a feature into Photoshop within a year of us seeing that other software.
This might be a long shot, but did the demo include zooming in on a picture of a skull to show a message written on one of the teeth? If so, I've been trying to find a video of it for years.
No, it was a photo of a building. The "demo" was the guy right there playing with the software, no video.
Perhaps, but size of photos should affect load times when starting the app (and in my opinion, so shouldn’t most feature either, but that depends on your architecture I suppose).
> Size of photos grown exponentially as well.
Photoshop can grow them itself and the below looks no and thread are amazing. https://news.ycombinator.com/item?id=26448986
Photoshop can grow them itself and the below looks no and thread are amazing. https://news.ycombinator.com/item?id=26448986
Yeah Jonathan Blow isn't exactly a luminary in computer science. I once read him going on a meltdown over linux ports because "programming is hard". This is the kind of minds Apple enable, i.e. "why isn't this easy?"
The entire point of computers is to make things easy.
The Apple question "why isn't this easy" is missing from 95% of UX in modern software. Stop acting as if software devs are the only users of software. And even then: computers should do work and not produce work or mental overhead.
The Apple question "why isn't this easy" is missing from 95% of UX in modern software. Stop acting as if software devs are the only users of software. And even then: computers should do work and not produce work or mental overhead.
> The Apple question "why isn't this easy" is missing from 95% of UX in modern software.
I switched to a MacBook at work a few months ago, and it's been an epic of frustration and Googling: e.g.,
1. I set the keyboard layout to UK PC, but it kept switching back to default. (I was accidentally mashing a keyboard shortcut designed to do just that. I've since disabled it.)
2. Clicking on a link or other web-page element will, occasionally and apparently randomly, take me back a page or two in my history rather than opening the link or activating the element. (At least in Chrome: I've not used another browser on macOS yet.)
3. Command-tabbing to a minimised application does not, as one would naively expect, automatically unminimise it. Instead I'm left staring at an apparently unchanged screen.
4. If I open Finder in a given folder, there is, as best can tell, no easy way to navigate to that folder's parent.
Now arguably #1 was the fault of my own ignorance (though some kind of obvious feedback as to what was happening and why would have been nice), and #2 may be down to Google rather than Apple.
But #3 and #4 are plain bad design, bordering on user-hostile.
So far I'm not seeing that Apple's reputation for superior UX is justified, at least not in laptops.
I switched to a MacBook at work a few months ago, and it's been an epic of frustration and Googling: e.g.,
1. I set the keyboard layout to UK PC, but it kept switching back to default. (I was accidentally mashing a keyboard shortcut designed to do just that. I've since disabled it.)
2. Clicking on a link or other web-page element will, occasionally and apparently randomly, take me back a page or two in my history rather than opening the link or activating the element. (At least in Chrome: I've not used another browser on macOS yet.)
3. Command-tabbing to a minimised application does not, as one would naively expect, automatically unminimise it. Instead I'm left staring at an apparently unchanged screen.
4. If I open Finder in a given folder, there is, as best can tell, no easy way to navigate to that folder's parent.
Now arguably #1 was the fault of my own ignorance (though some kind of obvious feedback as to what was happening and why would have been nice), and #2 may be down to Google rather than Apple.
But #3 and #4 are plain bad design, bordering on user-hostile.
So far I'm not seeing that Apple's reputation for superior UX is justified, at least not in laptops.
Window switching in OS X is so unintuitive it drives me MAD when you are remoting into a Mac.
Finder is just god awful and does all it can to obscure your actual filesystem location but the go to folder (cmd+g?) can get you where you need to go.
Finder is just god awful and does all it can to obscure your actual filesystem location but the go to folder (cmd+g?) can get you where you need to go.
The Apple reputation was absolutely justified back in the OS9 era, and the early iPhone as well. However both OS X and iOS7 and beyond were huge steps backwards in usability.
At this point I think Apple still deserves their reputation for superior UX, however that's a result of how epically bad Google, MS, and Facebook are at UX, not Apple doing a great job like they used to.
At this point I think Apple still deserves their reputation for superior UX, however that's a result of how epically bad Google, MS, and Facebook are at UX, not Apple doing a great job like they used to.
for 4. ...as best can tell, no easy way to navigate to that folder's parent
You can add a button to the toolbar in Finder (customize it from options) that when dropped down will show the complete path to the current folder as a list. You can use that to move up the tree.
You can add a button to the toolbar in Finder (customize it from options) that when dropped down will show the complete path to the current folder as a list. You can use that to move up the tree.
Cmd-Up brings you to the parent folder.
You can also cmd + click the current directory name in the menu bar to get the same menu.
Are you implying Apple software and devices are "easy"?
I think that's marketing, and not borne out of real experience.
Try setting up a new Apple device, there is very little in IT that is as frustrating and confusing.
I think that's marketing, and not borne out of real experience.
Try setting up a new Apple device, there is very little in IT that is as frustrating and confusing.
>Try setting up a new Apple device, there is very little in IT that is as frustrating and confusing.
It's a 5 minute process, and there's very little to it that it's not optimized for ease. Any champ can do it, and millions do.
Not sure what you're on about.
>I think that's marketing, and not borne out of real experience.
Yes, all those people are delluded.
It's a 5 minute process, and there's very little to it that it's not optimized for ease. Any champ can do it, and millions do.
Not sure what you're on about.
>I think that's marketing, and not borne out of real experience.
Yes, all those people are delluded.
Clearly you've never tried it, because it's certainly not 5 minutes, it's optimized for selling you bullshit Apple services and it's buggy as hell, with no feedback to the user why everything is broken and why you're having to re-authenticate five times in a row.
And good luck if you're setting up a family account for several devices with different iOS versions. You're gonna really need it.
And good luck if you're setting up a family account for several devices with different iOS versions. You're gonna really need it.
>Clearly you've never tried it, because it's certainly not 5 minutes, it's optimized for selling you bullshit Apple services and it's buggy as hell
I've tried it tons of times, have over 20 iOS/macOS devices over the years, and for some pervese reason, on macOS/OS X I like to install every major update on a clean disk too (and then re-import my data. It's an old Windows 95/XP-era reflex), so I do it at least once every year for my main driver (plus different new iOS devices).
What part of this sounds difficult?
https://www.youtube.com/watch?v=70u0x8Kf6j4
And the whole "optimized for selling you bullshit Apple services" is a couple of screens you can skip with one click -- and you might want to legitimately use too.
I've tried it tons of times, have over 20 iOS/macOS devices over the years, and for some pervese reason, on macOS/OS X I like to install every major update on a clean disk too (and then re-import my data. It's an old Windows 95/XP-era reflex), so I do it at least once every year for my main driver (plus different new iOS devices).
What part of this sounds difficult?
https://www.youtube.com/watch?v=70u0x8Kf6j4
And the whole "optimized for selling you bullshit Apple services" is a couple of screens you can skip with one click -- and you might want to legitimately use too.
Honestly, literally millions of people do this every year, and for most of them, it's like 10 minutes, plus the time waiting for the iCloud restore. Even my dad was able to set up his new iPad, and he's as techophobic as it gets.
Technophobic people are exactly the target audience that has a huge tolerance for broken software built on piles of abusive bullshit.
My father has very little patience for broken things, software or otherwise, so I'm really not sure what you're talking about.
It is definitly easier than configuring any GNU/Linux distribution.
Watching the video that coldtea posted, no, it is not. Ubuntu and most of its derivatives have very easy installers, and take a fraction of the time. The video didn't even include the time it would take to read and understand all the terms and conditions!
Assuming one wants a partial working computer after installation.
>This is the kind of minds Apple enable, i.e. "why isn't this easy?"
So the kind of minds we want?
"Why isn't this easy?" should be the bread and butter question of a programmer...
So the kind of minds we want?
"Why isn't this easy?" should be the bread and butter question of a programmer...
Except for problems that are hard. I wholeheartedly disagree, Blow showed many times he has not the right mindset.
> This is the kind of minds Apple enable, i.e. "why isn't this easy?"
I dunno, personally that's why I've used Apple products for the past decade, and I think it's also maybe part of why they have a 2T market cap, and are the most liquid publicly traded stock in the world?
I dunno, personally that's why I've used Apple products for the past decade, and I think it's also maybe part of why they have a 2T market cap, and are the most liquid publicly traded stock in the world?
So making simplistic products that treat users as dumb is profitable, yeah, I agree with that.
Believe it or not, lots of people have more important things to do with their computers than dick with them to make shit work.
Most people don't really know how their car works, and shouldn't have to. Same goes here.
Most people don't really know how their car works, and shouldn't have to. Same goes here.
Where's your evidence for that? The argument that it "treats users as dumb" and that doing so is "profitable" is oft trotted out, but I never see any substantiation for it. Plenty of companies do that. What's so special about Apple, then? I mean, it's gotta be something.
You gotta be careful about these arguments. They often have a slippery slope to a superiority complex (of "leet" NIX users over the unwashed "proles") hiding deep within.
Needlessly complex or powerful user interfaces aren't necessarily good. They were quite commonplace before Apple. Apple understood the value of minimalism, of cutting away interaction noise until there's nothing left to subtract. Aesthetically speaking, this is approach has a long, storied history with respect to mechanical design. It's successful because it works.
What Apple understood really acutely and mastered is human interface design. They perfected making human centric interfaces that look and feel like fluid prosthetic extensions of one's body, rather than computational interface where power is achieved by anchoring ones tasks around the machine for maximal efficiency. Briefly, they understood intuition. Are you arguing that intuition is somehow worse than mastering arcana, simple because you've done the latter?
Now, I'm not going to say that one is better than the other. I love my command line vim workflow dearly, and you'll have to pry my keyboard out of my cold dead hands. But there's definitely the idea of "right tool for the right job" that you might be sweeping by here. Remember, simplicity is as much a function of cherished *NIX tools you probably know and love. It's where they derive their power. Be careful of surface level dismissals (visual interfaces versus textual) that come from tasting it in a different flavor. You might miss the forest for the trees!
You gotta be careful about these arguments. They often have a slippery slope to a superiority complex (of "leet" NIX users over the unwashed "proles") hiding deep within.
Needlessly complex or powerful user interfaces aren't necessarily good. They were quite commonplace before Apple. Apple understood the value of minimalism, of cutting away interaction noise until there's nothing left to subtract. Aesthetically speaking, this is approach has a long, storied history with respect to mechanical design. It's successful because it works.
What Apple understood really acutely and mastered is human interface design. They perfected making human centric interfaces that look and feel like fluid prosthetic extensions of one's body, rather than computational interface where power is achieved by anchoring ones tasks around the machine for maximal efficiency. Briefly, they understood intuition. Are you arguing that intuition is somehow worse than mastering arcana, simple because you've done the latter?
Now, I'm not going to say that one is better than the other. I love my command line vim workflow dearly, and you'll have to pry my keyboard out of my cold dead hands. But there's definitely the idea of "right tool for the right job" that you might be sweeping by here. Remember, simplicity is as much a function of cherished *NIX tools you probably know and love. It's where they derive their power. Be careful of surface level dismissals (visual interfaces versus textual) that come from tasting it in a different flavor. You might miss the forest for the trees!
It's easy to take a stab at someone online, behind a keyboard, but I'd suggest you show us all your work and we'll judge your future opinions based on it.
By no metric I am comparatively as successful as the guy, but I am still able to disagree on his point that Linux is held back by its tools. The fact that he did not want or had the time to learn Linux's tooling don't mean anything in particular except that he's either very busy or very lazy. Any interview I read with him he's just ranting and crying over this or that "too complex" matter.
If he does not want to deal with the complexity of modern computers, he should design and build board games.
As an example, read how the one-man-band of Execution Unit manages to, in house, write and test his game on three operating systems. https://www.executionunit.com/blog/2019/01/02/how-i-support-...
It's not a matter of "this is too hard", Blow just does not want to do it, let's be honest.
As an example, read how the one-man-band of Execution Unit manages to, in house, write and test his game on three operating systems. https://www.executionunit.com/blog/2019/01/02/how-i-support-...
It's not a matter of "this is too hard", Blow just does not want to do it, let's be honest.
preferably future work and dreams and ramblings too..
preferably over a twitch stream..
preferably over a twitch stream..
If I wanted that experience I would kept my PC with MS-DOS 3.3.
I think there are two things that cause this.
One is that there’s a minimum performance that people will tolerate. Beyond that you get quickly diminishing user satisfaction returns when trying to optimize. The difference between 30 seconds and 10 seconds in app startup time isn’t going to make anyone choose or not choose Photoshop. People who use PS a lot probably keep it open all day and everyone else doesn’t care enough about the 20 seconds.
The second problem is that complexity scales super-linearly with respect to feature grown because each feature interacts with every other feature. This means that the difficulty of optimizing startup times gets harder as the application grows in complexity. No single engineer or team of engineers could fix the problem at this point, it would have to be a mandate from up high, which would be a silly mandate since the returns would likely be very small.
One is that there’s a minimum performance that people will tolerate. Beyond that you get quickly diminishing user satisfaction returns when trying to optimize. The difference between 30 seconds and 10 seconds in app startup time isn’t going to make anyone choose or not choose Photoshop. People who use PS a lot probably keep it open all day and everyone else doesn’t care enough about the 20 seconds.
The second problem is that complexity scales super-linearly with respect to feature grown because each feature interacts with every other feature. This means that the difficulty of optimizing startup times gets harder as the application grows in complexity. No single engineer or team of engineers could fix the problem at this point, it would have to be a mandate from up high, which would be a silly mandate since the returns would likely be very small.
If you haven't come across it, you might appreciate this post: https://adamdrake.com/command-line-tools-can-be-235x-faster-...
I'll give it a read. Thank you.
(I like anything that echos well in this chamber of mine... just kidding ;)
(I like anything that echos well in this chamber of mine... just kidding ;)
The problem is that if everything is simple enough. How to set your goals this year? Complication creates lots of jobs and waste. This makes us not starved but others somewhere in the world or in the future when resource all gone.
I see where you're coming from with this, but this is getting into the realm of social economics and thus politics.
To solve the problem you're describing we need to be better at protecting all members of society not in (high paying) jobs, such as universal basic income and, I don't know, actually caring about one another.
But I do see your point, and it's an interesting one to raise.
To solve the problem you're describing we need to be better at protecting all members of society not in (high paying) jobs, such as universal basic income and, I don't know, actually caring about one another.
But I do see your point, and it's an interesting one to raise.
"But developer productivity!"
Most orgs (and most devs) feel developer productivity should come first. They're not willing to (and in a lot of cases, not able to) optimize the apps they write. When things get hard (usually about 2 years in) devs just move on to the next job.
Most orgs (and most devs) feel developer productivity should come first. They're not willing to (and in a lot of cases, not able to) optimize the apps they write. When things get hard (usually about 2 years in) devs just move on to the next job.
> If people (developers/engineers) would only sit back...*
It is a matter of incentives. At many companies, developers are rewarded for shipping, and not for quality, efficiency, supportability, documentation, etc.* This is generally expected of a technology framework still in the profitability growth stage; once we reach a more income-oriented stage, those other factors will enter incentives to protect the income.
It is a matter of incentives. At many companies, developers are rewarded for shipping, and not for quality, efficiency, supportability, documentation, etc.* This is generally expected of a technology framework still in the profitability growth stage; once we reach a more income-oriented stage, those other factors will enter incentives to protect the income.
Maybe. I'm not convinced.
I think you can build something complex quickly and well at the same time. I built opskit.io in three weeks. It's about 90% automated.
I think you can build something complex quickly and well at the same time. I built opskit.io in three weeks. It's about 90% automated.
> I think you can build something complex quickly and well at the same time.
One definitely can. It's a real crapshoot whether the average developer can. For what it's worth, I consider myself a below-average developer. There is no way I could grind l33tcode for months and even land a FAANG interview. I can't code a red-black tree to save my life unless I had a textbook in front of me. Code I build takes an enormous amount of time to deliver, and So. Much. Searching. You get the picture. I'm a reasonably good sysadmin/consultant/sales-engineer; all other roles I put in ludicrous amounts of effort into, to become relevant. Good happenstance I enjoy the challenge.
For the time being however, there is such enormous demand for any talent that I always find myself in situations where my below average skills are treated as a scarcity. Like near-100 headcount testing organization in a tech-oriented business with an explicit leadership mandate to automate with developer-written integration code from that organization...and two developers, both with even worse skills than mine. When a developer balks at writing a single regular expression to insert a single character into the front of an input string, that's nearly the definition of turning one wrench for ten years; while I'm slow and very-not-brilliant, I'm a smart enough bear to look up how to do it on different OS' or languages and implement it within the hour.
This is not unusual in our industry. That's why FizzBuzz exists. That's just to clear the bar of someone who knows the difference between a hash and a linked list.
To clear the bar of "something complex quickly and well at the same time" though, I've found it insufficient to clear only the technical hurdle and obtain consistent results. The developer has to care about all the stakeholders. Being able to put themselves into the shoes of the future developers maintaining the codebase, future operators who manage first line support, future managers who seek summarized information about the state and history of the platform, future users who apply business applications to the platform, future support engineers feeding support results back into developers, and so on. That expansive, empathetic orientation to balance trade-offs and nuances is either incentivized internally, or staffed at great expense externally with lots of project coordination (though really, you simply kick the can upstairs to Someone With Taste Who Cares).
I'd sure as hell like to know alternatives that are repeatable, consistently-performing, and sustainable though. Closest I can think of is long-term apprenticeship-style career progression, with a re-dedicated emphasis upon staffing out highly-compensated technical writers, because I strongly suspect as an industry we're missing well-written story communication to tame the complexity monster; but that's a rant for another thread.
One definitely can. It's a real crapshoot whether the average developer can. For what it's worth, I consider myself a below-average developer. There is no way I could grind l33tcode for months and even land a FAANG interview. I can't code a red-black tree to save my life unless I had a textbook in front of me. Code I build takes an enormous amount of time to deliver, and So. Much. Searching. You get the picture. I'm a reasonably good sysadmin/consultant/sales-engineer; all other roles I put in ludicrous amounts of effort into, to become relevant. Good happenstance I enjoy the challenge.
For the time being however, there is such enormous demand for any talent that I always find myself in situations where my below average skills are treated as a scarcity. Like near-100 headcount testing organization in a tech-oriented business with an explicit leadership mandate to automate with developer-written integration code from that organization...and two developers, both with even worse skills than mine. When a developer balks at writing a single regular expression to insert a single character into the front of an input string, that's nearly the definition of turning one wrench for ten years; while I'm slow and very-not-brilliant, I'm a smart enough bear to look up how to do it on different OS' or languages and implement it within the hour.
This is not unusual in our industry. That's why FizzBuzz exists. That's just to clear the bar of someone who knows the difference between a hash and a linked list.
To clear the bar of "something complex quickly and well at the same time" though, I've found it insufficient to clear only the technical hurdle and obtain consistent results. The developer has to care about all the stakeholders. Being able to put themselves into the shoes of the future developers maintaining the codebase, future operators who manage first line support, future managers who seek summarized information about the state and history of the platform, future users who apply business applications to the platform, future support engineers feeding support results back into developers, and so on. That expansive, empathetic orientation to balance trade-offs and nuances is either incentivized internally, or staffed at great expense externally with lots of project coordination (though really, you simply kick the can upstairs to Someone With Taste Who Cares).
I'd sure as hell like to know alternatives that are repeatable, consistently-performing, and sustainable though. Closest I can think of is long-term apprenticeship-style career progression, with a re-dedicated emphasis upon staffing out highly-compensated technical writers, because I strongly suspect as an industry we're missing well-written story communication to tame the complexity monster; but that's a rant for another thread.
Has Jonathan Blow actually measured these things or is this perhaps a skewed memory?
Reminded me of Fabrice Bellard's Pi digits record[1]
The previous Pi computation record of about 2577 billion decimal digits was published by Daisuke Takahashi on August 17th 2009. The main computation lasted 29 hours and used 640 nodes of a T2K Open Supercomputer (Appro Xtreme-X3 Server). Each node contains 4 Opteron Quad Core CPUs at 2.3 GHz, giving a peak processing power of 94.2 Tflops (trillion floating point operations per second).
My computation used a single Core i7 Quad Core CPU at 2.93 GHz giving a peak processing power of 46.9 Gflops. So the supercomputer is about 2000 times faster than my computer. However, my computation lasted 116 days, which is 96 times slower than the supercomputer for about the same number of digits. So my computation is roughly 20 times more efficient.
[1]: https://bellard.org/pi/pi2700e9/faq.html
The previous Pi computation record of about 2577 billion decimal digits was published by Daisuke Takahashi on August 17th 2009. The main computation lasted 29 hours and used 640 nodes of a T2K Open Supercomputer (Appro Xtreme-X3 Server). Each node contains 4 Opteron Quad Core CPUs at 2.3 GHz, giving a peak processing power of 94.2 Tflops (trillion floating point operations per second).
My computation used a single Core i7 Quad Core CPU at 2.93 GHz giving a peak processing power of 46.9 Gflops. So the supercomputer is about 2000 times faster than my computer. However, my computation lasted 116 days, which is 96 times slower than the supercomputer for about the same number of digits. So my computation is roughly 20 times more efficient.
[1]: https://bellard.org/pi/pi2700e9/faq.html
I once joked with a colleague that my sqlite3 install is faster than his Hadoop cluster for running a report across a multi-gig file.
We benchmarked it, i was much, much faster.
Technically though once that multi-gig file becomes many hundreds of gigs, my computer would loose by a huge margin.
We benchmarked it, i was much, much faster.
Technically though once that multi-gig file becomes many hundreds of gigs, my computer would loose by a huge margin.
"Command-line Tools can be 235x Faster than your Hadoop Cluster"
https://web.archive.org/web/20200414235857/https://adamdrake...
https://web.archive.org/web/20200414235857/https://adamdrake...
I recently did some data processing on a single (albeit beefy node) someone had been using a cluster for. I composed and ran ETL in a day what took them weeks in their infrastructure (they were actually still in the process of fixing it).
At that point you can just get a bigger computer though.
No you cannot, you cannot infinitely scale SQLite, you can’t load 100 G of data into a single SQLite file in any meaningful amount of time. Then try creating an index on it and cry.
I have tried this, I literally wanted to create a simple web app that is powered by the cheapest solution possible, but it had to serve from a database that cannot be smaller than 150GB. SQLite failed. Even Postgres by itself was very hard! In the end I now launch redshift for a couple days, process all the data, then pipe it to Postgres running on a lightsail vps via dblink. Haven’t found a better solution.
I have tried this, I literally wanted to create a simple web app that is powered by the cheapest solution possible, but it had to serve from a database that cannot be smaller than 150GB. SQLite failed. Even Postgres by itself was very hard! In the end I now launch redshift for a couple days, process all the data, then pipe it to Postgres running on a lightsail vps via dblink. Haven’t found a better solution.
My rule of thumb is that a single processor core can handle about 100MB/s, if using the right software (and using the software right). For simple tasks, this kan be 200+ MB/s, if there is a lot of random access (both against memory and against storage), one can assume about 10k-100k IOPS per core.
For a 32 core processor, that means that it can process a data set of 100G in the order of 30 seconds. For some types of tasks, it can be slower, and if the processing is either light or something that lets you leverage specialized hardware (such as a GPU), it can be much faster. But if you start to take hours to process a dataset of this size (and you are not doing some kind of heavy math), you may want to look at your software stack before starting to scale out. Not only to save on hardware resources, but also because it may require less of your time to optimize a single node than to manage a cluster.
For a 32 core processor, that means that it can process a data set of 100G in the order of 30 seconds. For some types of tasks, it can be slower, and if the processing is either light or something that lets you leverage specialized hardware (such as a GPU), it can be much faster. But if you start to take hours to process a dataset of this size (and you are not doing some kind of heavy math), you may want to look at your software stack before starting to scale out. Not only to save on hardware resources, but also because it may require less of your time to optimize a single node than to manage a cluster.
> "using the right software (and using the software right)"
This is a great phrase that I'm going to use more.
This is a great phrase that I'm going to use more.
This is a great rule of thumb which helps build a kind of intuition around performance I always try to have my engineers contextualizing. The "lazy and good" way (which has worked I'd say at least 9/10 times in my career when I run into these problems) is to find a way to reduce data cardinality ahead of intense computation. It's 100% for the reason you describe in your last sentence -- it doesn't just save on hardware resources, but it potentially precludes any timespace complexity bottlenecks from becoming your pain point.
>No you cannot, you cannot infinitely scale SQLite, you can’t load 100 G of data into a single SQLite file in any meaningful amount of time. Then try creating an index on it and cry.
Yes, you can. Without indexes to slow you down (you can create them afterwards), it isn't even much different than any other DB, if not faster.
>Even Postgres by itself was very hard!
Probably depends on your setup. I've worked with multi-TB sized Postgres single databases (heck, we had 100GB in a single table without partitions). Then again the machine had TB sized RAM.
Yes, you can. Without indexes to slow you down (you can create them afterwards), it isn't even much different than any other DB, if not faster.
>Even Postgres by itself was very hard!
Probably depends on your setup. I've worked with multi-TB sized Postgres single databases (heck, we had 100GB in a single table without partitions). Then again the machine had TB sized RAM.
> but it had to serve from a database that cannot be smaller than 150GB. SQLite failed. Even Postgres by itself was very hard!
The PostgreSQL database for a CMS project I work on weighs about 250GB (all assets are binary in the database), and we have no problem at all serving a boatload of requests (with the replicated database and the serving CMS running on each live server, with 8GB of RAM).
To me, it smells like you've lacked some indices or ran on a rpi?
The PostgreSQL database for a CMS project I work on weighs about 250GB (all assets are binary in the database), and we have no problem at all serving a boatload of requests (with the replicated database and the serving CMS running on each live server, with 8GB of RAM).
To me, it smells like you've lacked some indices or ran on a rpi?
It sounds like the op is trying to provision and load 150GB in a reasonably fast manner. Once loaded, presumably any of the usual suspects will be fast enough. It’s the up front loading costs which are the problem.
Anyway, I’m curious what kind of data the op is trying to process.
Anyway, I’m curious what kind of data the op is trying to process.
I am trying to load and serve the Microsoft academic graph to produce author profile pages for all academic authors! Microsoft and google already do this but IMO they leave a lot to be desired.
But this means there are a hundred million entities, publishing 3x number of papers and a bunch of metadata associated. On redshift I can get all of this loaded in minutes and takes like 100G but Postgres loads are pathetic comparatively.
And I have no intention of spending more than 30 bucks a month! So hard problem for sure! Suggestions welcome!
But this means there are a hundred million entities, publishing 3x number of papers and a bunch of metadata associated. On redshift I can get all of this loaded in minutes and takes like 100G but Postgres loads are pathetic comparatively.
And I have no intention of spending more than 30 bucks a month! So hard problem for sure! Suggestions welcome!
There are settings in Postgres that allow for bulk loading.
By default you get a commit after each INSERT which slows things down by a lot.
By default you get a commit after each INSERT which slows things down by a lot.
How many rows are we talking about? In the end once I started using dblink to load via redshift after some preprocessing the loads were reasonable, and indexing too. But I’m looking at full data refreshes every two weeks and a tight budget (30 bucks a month) so am constrained on solutions. Suggestions welcome!
Try DuckDB! I've been getting 20x SQLite performance on one thread, and it usually scales linearly with threads!
Maybe I'm misunderstanding, but this seems very strange. Are you suggesting that Postgres can't handle a 150GB database with acceptable performance?
I’m trying to run a Postgres instance on a basic vps instance with a single vcpu and 8gb of ram! And I’ll need to erase and reload all 150 GB every two weeks..
Had a similar problem recently. Ended up creating a custom system using a file-based index (append to files named by the first 5 char of the SHA1 of the key)
Took 10 hours to parse my Terabyte. Uploaded it to Azure Blob storage, now I can query my 10B rows in 50ms for ~10^-7$.
It's hard to evolve, but 10x faster and cheaper than other solutions.
Curious if you tried this on an EC2 instance in AWS? The IOPS for EBS volume are notoriously low, and possibly why a lot of self-hosted DB instances feel very slow vs similarly priced AWS services.
Personal anecdote, but moving to a a dedicated server from EC2 increased the max throughput by a factor of 80 for us.
Did you try to compare that to EC2 instances with ephemeral nvme drives? I'm seeing hdfs throughput of up to several GB/node using such instances.
You can use locally attached SSD instances. Then you're responsible for its reliability so not getting all the 'cloud' benefits. Used them for provisioning own CI cluster with raid-0 btrfs running PostgreSQL. Only backed up the provisioning and CI scripts.
Got burned there for sure! Speed is one thing but the cost is outrageous for io heavy apps! Anyways I moved to lightsail which doesn’t have io costs paradoxically so while io is slow at least the cost is predictable!
A couple of 64gb ram sticks and you can fit your data in ram.
It's all about the right tool for the job('s scale.)
You can skip Hadoop and go from SQLite to something like S3 + Presto that scales to extremely high volumes with low latency and better than linear financial scaling.
Does hundreds of gigs introduce a general performance hit or could it still be further optimized using some smart indexing strategy?
Everything is fast if it fits in memory and caches.
Unless it's accidentally quadratic. Then all the RAM in the world isn't going to help you.
And in 2021, almost everything* does.
*Obviously not really. But very very many things do, even doing useful jobs in production, as long as you have high enough specs.
*Obviously not really. But very very many things do, even doing useful jobs in production, as long as you have high enough specs.
[deleted]
I've had similar experiences. Sometimes we'll have a dataset with tens of thousands records and it will give rise to the belief that it's a problem that requires a highly scalable solution because "tens of thousands" is more than a human can hold in their head. In reality, if the records are just a few columns of data, the whole set can be serialized to a single file and consumed in one gulp into a single object in memory on commodity hardware no sweat. Then process it with a for loop. Very few enterprises actually have big big data.
My solution started out as a 10-line Python script where I would manually clean the data we received, then process it.
CEO: "Will this scale?"
Me: "No, absolutely not, at some point we'll need to hire someone who knows what they're doing."
As time passed and we got more data, I significantly improved the data cleaning portions so that most of it was automated, and the parts that weren't automated would be brought up as suggestions I could quickly handle. I learned the very basics of performance and why `eval` is bad, set up my script so I didn't need to hard-code the number of files to process each day, started storing data on a network drive and then eventually a db...
I still don't know what I'm doing, but by the time I left it took maybe 5 minutes of manual data cleaning to handle thousands of jobs a day, and then the remainder could be done on a single machine.
Me: "No, absolutely not, at some point we'll need to hire someone who knows what they're doing."
As time passed and we got more data, I significantly improved the data cleaning portions so that most of it was automated, and the parts that weren't automated would be brought up as suggestions I could quickly handle. I learned the very basics of performance and why `eval` is bad, set up my script so I didn't need to hard-code the number of files to process each day, started storing data on a network drive and then eventually a db...
I still don't know what I'm doing, but by the time I left it took maybe 5 minutes of manual data cleaning to handle thousands of jobs a day, and then the remainder could be done on a single machine.
Said enterprise WISH they had big data. Or maybe it's fear, as in, 'what IF we get big data?'
I'm aware of a couple of companies who behave like that - "well, we could increase our user based by an order of magnitude at any point here, so better spring for the order of magnitude more expensive database, just in case we need it."
Feels like toxic optimism.
Feels like toxic optimism.
It's not just about scaling databases, some people are simply unable to assess reasonable limits on any system. A few years ago certain Scandinavian publisher decided to replace their standard industry tools by a single "Digital Experience Platform" that was expected to do everything. After a couple of years they understood it's a stupid idea and gave up. Then later someone in the management thought that since they already spent some millions of euros they should continue anyway. This behemoth is so slow and buggy the end users work at 1/4th speed but everyone is afraid to say anything as the ones who did have been fired. The current PM is sending weekly success messages. It's hilarious. And all because someone once had a fantasy of having one huge system that does everything.
I love the phrase 'toxic optimism'.
I've noticed business people have a different idea of what 'big data' means to tech guys. The business guys think it means a lot of data like the records of a million people which is a lot of data but not the tech guy definition which tends to be data too large to process on a single machine.
Those come out at something like 1GB and 10TB which are obviously rather different.
Those come out at something like 1GB and 10TB which are obviously rather different.
Unfortunately, this kind of behavior will be rewarded by the job market, because he's now got a bunch more tech buzzwords on his resume than you. Call it the Resume Industrial Complex: engineers build systems with as many bells and whistles as possible, because they want to learn all the hot new tech stacks so they can show off their extensive "skills" to potential employers.
I wonder what percentage of data center power is wasted running totally unnecessary trendy abstractions...
God I love stories like this
My favorite part of conducting design interviews is when a candidate has pulled some complex distributed system out of their ass, and I ask them what the actual throughput/memory usage looks like.
And what would happen the day you were hit by a bus?
On that day, most probably nothing with regards to this task.
Then, later, probably someone would check out the scripts from a shared repo. Then, read an outdated README, try it out, swear a bit, check for correctness with someone dependent on the results, and finally learn how to do the task.
There is a lot of business processes that can tolerate days or weeks of delay in case of such a tragic (and hopefully improbable) event. The trick is to know which of them can't.
Then, later, probably someone would check out the scripts from a shared repo. Then, read an outdated README, try it out, swear a bit, check for correctness with someone dependent on the results, and finally learn how to do the task.
There is a lot of business processes that can tolerate days or weeks of delay in case of such a tragic (and hopefully improbable) event. The trick is to know which of them can't.
That second paragraph is very relatable.
> There is a lot of business processes that can tolerate days or weeks of delay in case of such a tragic (and hopefully improbable) event. The trick is to know which of them can't.
This is really true BUT that kind of problems are OK - nobody cares - until somebody starts caring and then all of a sudden it is urgent (exactly because they were undetected for weeks/months due to their periodicity)
This is really true BUT that kind of problems are OK - nobody cares - until somebody starts caring and then all of a sudden it is urgent (exactly because they were undetected for weeks/months due to their periodicity)
I meant it's fine to take calculated risks.
E. g. we have less than 1% chance per year that a given person leaves us on bad terms or suffers a bad accident or illness. In case it really happens, it will cost us X in delays and extra work. To lower the probability of this risk to Y% would cost us Z (money, delay, etc).
If you do this math, you can tell if it's a good idea to optimize here, or if you have more pressing issues.
In my experience, this sort of one-man jobs gets automated or at least well described and checked for fear of mistakes and/or employee fraud rather than "downtime".
E. g. we have less than 1% chance per year that a given person leaves us on bad terms or suffers a bad accident or illness. In case it really happens, it will cost us X in delays and extra work. To lower the probability of this risk to Y% would cost us Z (money, delay, etc).
If you do this math, you can tell if it's a good idea to optimize here, or if you have more pressing issues.
In my experience, this sort of one-man jobs gets automated or at least well described and checked for fear of mistakes and/or employee fraud rather than "downtime".
Mistakes are "downtime" as well in a way. Or maybe better, downtime is a mistake, cause errors and lead to problems.
Another guy install Postgres on his machine, runs a git clone, connects to the VPN and initiates the jobs?
I wasn't even using a db at the time, it was 100% pandas. We did eventually set up more infrastructure, when I left the data was loaded into the company's SQL Server db, then pulled into pandas, then uploaded back into a different table.
It's true – at that point, if I had disappeared without providing any transition help, the company would have been in trouble for a few days. But that goes for any employee – we were only 7 people at the time!
Eventually I built out some more infrastructure to run the jobs automatically on a dedicated machine, but last I checked everything still runs on one instance.
Eventually I built out some more infrastructure to run the jobs automatically on a dedicated machine, but last I checked everything still runs on one instance.
All of my routine tasks are documented in the operations manual. I'd be missed but the work would still get done.
SO is always impressive - love that their redis servers with 256GB RAM peak at 2% CPU load :)
SO is also my go-to argument when some smart "architect" proposes redundant Kubernetes cluster instances for some company-local project. People seem to have lost the feeling what is needed to serve a couple of thousand concurrent users (For company internal usages which I specialize in, you hardly will get more users). Everyone thinks they are Google or Netflix. Meanwhile, SO runs on 1-2 Racks with an amount of server that would not even justify kubernetes or even docker.
SO is also my go-to argument when some smart "architect" proposes redundant Kubernetes cluster instances for some company-local project. People seem to have lost the feeling what is needed to serve a couple of thousand concurrent users (For company internal usages which I specialize in, you hardly will get more users). Everyone thinks they are Google or Netflix. Meanwhile, SO runs on 1-2 Racks with an amount of server that would not even justify kubernetes or even docker.
SO really isn't a great example, they have considerations most companies don't - Windows and SQL Server licensing. When shit like that is involved, scale out rarely seems like a better choice.
It's not only about the amount of users, it's also a matter of availability. Even the most stupid low-use barely-does-anything internal apps at my company get deployed either to two machines or a Nomad cluster for redundancy ( across two DCs). Design for failure and all that. Failure is unlikely, but it's trivial to setup at least active-passive redundancy just in case, it will make failures much easier.
It's not only about the amount of users, it's also a matter of availability. Even the most stupid low-use barely-does-anything internal apps at my company get deployed either to two machines or a Nomad cluster for redundancy ( across two DCs). Design for failure and all that. Failure is unlikely, but it's trivial to setup at least active-passive redundancy just in case, it will make failures much easier.
The "1-2 racks" perspective is great too, really makes you think the old XKCD joke [1] about tripping over the power cable might not be that far wrong. ;-)
[1] https://xkcd.com/908/
[1] https://xkcd.com/908/
> SO is also my go-to argument when some smart "architect" proposes redundant Kubernetes cluster instances for some company-local project.
Technically you don't need Kubernetes, yes. But: There are advantages that Kubernetes gives you even for a small shop:
- assuming you have a decent shared storage, it's a matter of about 30 minutes to replace a completely failed machine - plug the server in, install a bare-bones Ubuntu, kubeadm join, done. If you use Puppet and netboot install, you can go even faster (Source: been there, done that). And the best thing: assuming well written health checks users won't even notice you just had a node fail as k8s will take care of rescheduling.
- no need to wrangle with systemd unit files (or, worse, classic init.d scripts) for your application. For most scenarios you will either find Docker-embedded healthchecks somewhere or you can easily write your own so that Kubernetes can automatically
- no "hidden undocumented state" like wonky manual customizations somewhere in /etc that can mess up disaster recovery / horizontal scale, as everything relevant is included in either the Kubernetes spec or the Docker images. Side effect: this also massively reduces the ops load during upgrades, as all there is on a typical k8s node should be the base OS and Docker (or, in newest k8s versions, not even that anymore)
- it's easy to set up new development instances in a CI/CD environment
- generally, it's easier to get stuff done in corporate environments: just spin up a container on your cluster and that's it, no wrestling with finance and three levels of sign-off to get approval for a VM or, worse, bare metal.
I won't deny that there are issues though, especially if you're selfhosting:
- you will end up with issues with basic network tasks very quickly during setup, MetalLB is a nightmare, but smooth once you do have set it up. Most stuff is made with the assumption of every machine being in a fully Internet-reachable cluster (coughs in certbot), once you diverge from that (e.g. because of corp requiring you have to have dedicated "load balancer" nodes that only serve to direct traffic from outside to inside and "application" nodes not be directly internet-reachable) you're on your own.
- most likely you'll end up with one or two sandwich layers of load balancing (k8s ingress for one, and if you have it an external LB/WAF), which makes stuff like XFF headers ... interesting to say the least
- same if you're running anything with UDP, e.g. RTMP streaming
- the various networking layers are extremely hard to debug as most of k8s networking (no matter the overlay you use) is a boatload of iptables black magic. Even if you have a decade of experience...
Technically you don't need Kubernetes, yes. But: There are advantages that Kubernetes gives you even for a small shop:
- assuming you have a decent shared storage, it's a matter of about 30 minutes to replace a completely failed machine - plug the server in, install a bare-bones Ubuntu, kubeadm join, done. If you use Puppet and netboot install, you can go even faster (Source: been there, done that). And the best thing: assuming well written health checks users won't even notice you just had a node fail as k8s will take care of rescheduling.
- no need to wrangle with systemd unit files (or, worse, classic init.d scripts) for your application. For most scenarios you will either find Docker-embedded healthchecks somewhere or you can easily write your own so that Kubernetes can automatically
- no "hidden undocumented state" like wonky manual customizations somewhere in /etc that can mess up disaster recovery / horizontal scale, as everything relevant is included in either the Kubernetes spec or the Docker images. Side effect: this also massively reduces the ops load during upgrades, as all there is on a typical k8s node should be the base OS and Docker (or, in newest k8s versions, not even that anymore)
- it's easy to set up new development instances in a CI/CD environment
- generally, it's easier to get stuff done in corporate environments: just spin up a container on your cluster and that's it, no wrestling with finance and three levels of sign-off to get approval for a VM or, worse, bare metal.
I won't deny that there are issues though, especially if you're selfhosting:
- you will end up with issues with basic network tasks very quickly during setup, MetalLB is a nightmare, but smooth once you do have set it up. Most stuff is made with the assumption of every machine being in a fully Internet-reachable cluster (coughs in certbot), once you diverge from that (e.g. because of corp requiring you have to have dedicated "load balancer" nodes that only serve to direct traffic from outside to inside and "application" nodes not be directly internet-reachable) you're on your own.
- most likely you'll end up with one or two sandwich layers of load balancing (k8s ingress for one, and if you have it an external LB/WAF), which makes stuff like XFF headers ... interesting to say the least
- same if you're running anything with UDP, e.g. RTMP streaming
- the various networking layers are extremely hard to debug as most of k8s networking (no matter the overlay you use) is a boatload of iptables black magic. Even if you have a decade of experience...
Your arguments are true, but you did not consider the complexity that you have now introduced in a small shop operation. You will need kubernetes knowledge and experienced engineers on that matter. I would argue that the SO setup with 9 webservers, 2x2 DB servers and 2 redis servers could easily be administered with 20 year old knowledge about networks and linux/windows itself.
And I also argue, lack of experience of fiddling with redundant kubernetes is a more likely source of downtime than hardware failure and keeping things simple.
And I also argue, lack of experience of fiddling with redundant kubernetes is a more likely source of downtime than hardware failure and keeping things simple.
Thankfully in this case there is 13 years of knowledge in the DB itself.
> You will need kubernetes knowledge and experienced engineers on that matter.
For a small shop you'll need one person knowing that stuff, or you bring in an external consultant for setting up and maintaining the cluster, or you move to some cloud provider (k8s is basically a commodity that everyone and their dog offers, not just the big 3!) so you don't have to worry about that at all.
And a cluster for basic stuff is not even that expensive if you do want to run your own. Three worker machines and one (or, if you want HA, two) NAS systems... half a rack and you're set.
The benefit you have is your engineers will waste a lot less time setting up, maintaining and tearing down development and QA environments.
As for the SO setup: the day-to-day maintenance of them should be fairly simple - but AFAIK they had to do a lot of development effort to get the cluster to that efficiency, including writing their own "tag DB".
For a small shop you'll need one person knowing that stuff, or you bring in an external consultant for setting up and maintaining the cluster, or you move to some cloud provider (k8s is basically a commodity that everyone and their dog offers, not just the big 3!) so you don't have to worry about that at all.
And a cluster for basic stuff is not even that expensive if you do want to run your own. Three worker machines and one (or, if you want HA, two) NAS systems... half a rack and you're set.
The benefit you have is your engineers will waste a lot less time setting up, maintaining and tearing down development and QA environments.
As for the SO setup: the day-to-day maintenance of them should be fairly simple - but AFAIK they had to do a lot of development effort to get the cluster to that efficiency, including writing their own "tag DB".
You will always need 2-3 experts, because in case of an incident, your 1 engineer might be on sick/holiday leave.
Well but lets walk one step back looking at SO, they are a windows shop (.NET, MS SQL Server) so I doubt k8s would be found in their setup.
Well but lets walk one step back looking at SO, they are a windows shop (.NET, MS SQL Server) so I doubt k8s would be found in their setup.
Ah yes, I’ll make my critical infrastructure totally dependent on some outside consultant who may or may not be around when I really need him. That sounds like a great strategy. /s
SO is a great counter example to many over complicated setups, but they have a few important details going for them.
> Everytime you go to SO, it hits one of these 9 web servers
This isn't strictly true. Most SO traffic is logged out, most doesn't require strictly consistent data, most can be cached at the CDN. This means most page views should never reach their servers.
This is obviously a great design! Caching at the CDN is brilliant. But there are a lot of services that can't be built like this.
> Everytime you go to SO, it hits one of these 9 web servers
This isn't strictly true. Most SO traffic is logged out, most doesn't require strictly consistent data, most can be cached at the CDN. This means most page views should never reach their servers.
This is obviously a great design! Caching at the CDN is brilliant. But there are a lot of services that can't be built like this.
CDN caches static assets. The request still goes to SO servers. Search goes to one of their massive Elastic Search servers.
I’m not saying we should all use SO’s architecture, I am trying to shed light on what’s possible.
YMMV obviously.
I’m not saying we should all use SO’s architecture, I am trying to shed light on what’s possible.
YMMV obviously.
Are you an SO dev? I had thought I read about the use of CDNs and/or Varnish or something like that for rendered pages for logged out users? I don't want to correct you on your own architecture if you are!
No, not a dev at SO. I am guessing what would be rather a standard use of CDN (hosting static assets, caching them geographically).
What you're saying is probably right.
What you're saying is probably right.
We went all-in on vertical scaling with our product. We went so far we decided on SQLite because we were never going to plan to have a separate database server (or any separate host for that matter). 6 years later that assumption has still held very strong and yielded incredible benefits.
The slowest production environment we run in today is still barely touched by our application during the heaviest parts of the day. We use libraries and tools capable of pushing millions of requests per second, but we typically only demand tens to hundreds throughout the day.
Admitting your scale fits on a single host means you can leverage benefits that virtually no one else is even paying attention to anymore. These benefits can put entire sectors of our industry out of business if more developers were to focus on them.
The slowest production environment we run in today is still barely touched by our application during the heaviest parts of the day. We use libraries and tools capable of pushing millions of requests per second, but we typically only demand tens to hundreds throughout the day.
Admitting your scale fits on a single host means you can leverage benefits that virtually no one else is even paying attention to anymore. These benefits can put entire sectors of our industry out of business if more developers were to focus on them.
Do you have any more details on your application? Sounds like your architecture choice worked out really well. I'm curious to hear more about it.
Our technology choices for the backend are incredibly straightforward. The tricky bits are principally .NET Core and SQLite. One new technology we really like is Blazor, because their server-side mode of operation fits perfectly with our "everything on 1 server" grain, and obviates the need for additional front-end dependencies or APIs.
how do you handle backup/replication for your sqlite server?
I’m not the OP but I’m the author of an open source tool called Litestream[1] that does streaming replication of SQLite databases to AWS S3. I’ve found it to be a good, cheap way of keeping your data safe.
[1]: https://litestream.io/
[1]: https://litestream.io/
I am definitely interested in a streaming backup solution. Right now, our application state is scattered across many independent SQLite databases and files.
We would probably have to look at a rewrite under a unified database schema to leverage something like this (at least for the business state we care about). Streaming replication implies serialization of total business state in my head, and this has some implications for performance.
Also, for us, backup to the cloud is a complete non-starter. We would have to have our customers set up a second machine within the same network (not necessarily same building) to receive these backups due to the sensitive nature of the data.
What I really want to do is keep all the same services & schemas we have today, but build another layer on top so that we can have business services directly aware of replication concerns. For instance, I might want to block on some targeted replication activity rather than let it complete asynchronously. Then, instead of a primary/backup, we can just have 4-5 application nodes operating as a cluster with some sort of scheme copying important entities between nodes as required. We already moved to GUIDs for a lot of identity due to configuration import/export problems, so that problem is solved already. There are very few areas of our application that actually require consensus (if we had multiple participants in the same environment), so this is a compelling path to explore.
We would probably have to look at a rewrite under a unified database schema to leverage something like this (at least for the business state we care about). Streaming replication implies serialization of total business state in my head, and this has some implications for performance.
Also, for us, backup to the cloud is a complete non-starter. We would have to have our customers set up a second machine within the same network (not necessarily same building) to receive these backups due to the sensitive nature of the data.
What I really want to do is keep all the same services & schemas we have today, but build another layer on top so that we can have business services directly aware of replication concerns. For instance, I might want to block on some targeted replication activity rather than let it complete asynchronously. Then, instead of a primary/backup, we can just have 4-5 application nodes operating as a cluster with some sort of scheme copying important entities between nodes as required. We already moved to GUIDs for a lot of identity due to configuration import/export problems, so that problem is solved already. There are very few areas of our application that actually require consensus (if we had multiple participants in the same environment), so this is a compelling path to explore.
You can stream back ups of multiple database files with Litestream. Right now you have to explicitly name them in the Litestream configuration file but in the future it will support using a glob or file pattern to pick up multiple files automatically.
As for cloud backup, that's just one replica type. It's usually the most common so I just state that. Litestream also supports file-based backups so you could do a streaming backup to an NFS mount instead. There's an HTTP replica type coming in v0.4.0 that's mainly for live read replication (e.g. distribute your query load out to multiple servers) but it could also be used as a backup method.
As for synchronous replication, that's something that's on the roadmap but I don't have an exact timeline. It'll probably be v0.5.0. The idea is that you can wait to confirm that data is replicated before returning a confirmation to the client.
We have a Slack[1] as well as a bunch of docs on the site[2] and an active GitHub project page. I do office hours[3] every Friday too if you want to chat over zoom.
[1]: https://join.slack.com/t/litestream/shared_invite/zt-n0j4s3c...
[2]: https://litestream.io/
[3]: https://calendly.com/benbjohnson/litestream
As for cloud backup, that's just one replica type. It's usually the most common so I just state that. Litestream also supports file-based backups so you could do a streaming backup to an NFS mount instead. There's an HTTP replica type coming in v0.4.0 that's mainly for live read replication (e.g. distribute your query load out to multiple servers) but it could also be used as a backup method.
As for synchronous replication, that's something that's on the roadmap but I don't have an exact timeline. It'll probably be v0.5.0. The idea is that you can wait to confirm that data is replicated before returning a confirmation to the client.
We have a Slack[1] as well as a bunch of docs on the site[2] and an active GitHub project page. I do office hours[3] every Friday too if you want to chat over zoom.
[1]: https://join.slack.com/t/litestream/shared_invite/zt-n0j4s3c...
[2]: https://litestream.io/
[3]: https://calendly.com/benbjohnson/litestream
I really like what I am seeing so far. What is the rundown on how synchronous replication would be realized? Feels like I would have to add something to my application for this to work, unless we are talking about modified versions of SQLite or some other process hooking approach.
Litestream maintains a WAL position so it would need to expose the current local WAL position & the highest replicated WAL position via some kind of shared memory—probably just a file similar to SQLite's "-shm" file. The application can check the current position when a transaction starts and then it can block until the transaction has been replicated. That's the basic idea from a high level.
Our backup strategy is to periodically snapshot the entire host volume via relevant hypervisor tools. We have negotiated RPOs with all of our customers that allow for a small amount of data loss intraday (I.e. w/ 15 minute snapshot intervals, we might lose up to 15 minutes of live business state). There are other mitigating business processes we have put into place which bridge enough of this gap for it to be tolerable for all of our customers.
In the industry we work in, as long as your RTO/RPO is superior to the system of record you interface with, you are never the sore thumb sticking out of the tech pile.
In our 6-7 years of operating in this manner, we still have not had to restore a single environment from snapshot. We have tested it several times though.
You will probably find that VM snapshot+restore is a ridiculously easy and reliable way to provide backups if you put all of your eggs into one basket.
In the industry we work in, as long as your RTO/RPO is superior to the system of record you interface with, you are never the sore thumb sticking out of the tech pile.
In our 6-7 years of operating in this manner, we still have not had to restore a single environment from snapshot. We have tested it several times though.
You will probably find that VM snapshot+restore is a ridiculously easy and reliable way to provide backups if you put all of your eggs into one basket.
>> You will probably find that VM snapshot+restore is a ridiculously easy and reliable way to provide backups if you put all of your eggs into one basket.
Yep, this is something we rely on whenever we perform risky upgrades or migrations. Just snapshot the entire thing and restore it if something goes wrong, and it's both fast and virtually risk-free.
Yep, this is something we rely on whenever we perform risky upgrades or migrations. Just snapshot the entire thing and restore it if something goes wrong, and it's both fast and virtually risk-free.
Does your application run on your own servers, your customers' servers, or some of each? I gather from your comments that you deploy your application into multiple production environments, presumably one per customer.
Both. We run a QA instance for every customer in our infrastructure, and then 2-3 additional environments per customer in their infrastructure.
Vertical scaling maybe works forever for the 99% of companies that are CRUD apps running a basic website. As soon as you add any kind of 2D or 3D processing like image, video, etc. you pretty much have to have horizontal scaling at some point.
The sad truth is that your company probably won't be successful (statistically). You pretty never have to consider horizontal scaling until you have a few hundred thousand DAU.
The sad truth is that your company probably won't be successful (statistically). You pretty never have to consider horizontal scaling until you have a few hundred thousand DAU.
You don’t need to scale your application vertically even with media processing, you just need to distribute that chunk of the work, which is a lot easier (no state).
Like someone else said, distributing work across multiple machines is a form of horizontal scaling.
> Like someone else said, distributing work across multiple machines is a form of horizontal scaling.
Sure, but it is the easy kind, when it comes to images or videos. Lambda, for example, can handle a huge amount of image processing for pennies per month and there is none of the additional machine baggage that comes with traditional horizontal scaling.
Sure, but it is the easy kind, when it comes to images or videos. Lambda, for example, can handle a huge amount of image processing for pennies per month and there is none of the additional machine baggage that comes with traditional horizontal scaling.
It really depends. Handling streaming video service that does any kind of reprocessing of the data would probably be better off with horizontal scaling.
I imagine its still super simple to have one core app that handles most of the logic and then a job queue system that runs these high load jobs on worker machines.
Much simpler than having everything split.
Much simpler than having everything split.
Worker machines sound like horizontal scaling to me.
Sure but its a massive amount simpler than some massively distributed microservice app where every component runs on multiple servers.
Most of these vertical scaling examples given actually do use multiple servers but the core is one very powerful server.
Most of these vertical scaling examples given actually do use multiple servers but the core is one very powerful server.
Definitely. There is certainly a place for Horizontal scaling. Just wanted to highlight how underrated vertical scaling is and a good engineer would evaluate these scaling options with prudence and perspicacity, not cult behavior so often observed in software engineering circles.
I think somehow this is related to how business minded people think too. I went to a course where people learn to pitch their ideas to get funds but the basics of business simply did exist much among the technical people.
One simple example (which I suspect most of the business do) is that you do all work either manually yourself or on your laptop while advise them as a resource-rich service. Only when you truly can not handle the demand then you may 'scale up' and turn your business into 'real' business. And there are plenty of tricks like this (as legally as possible).
One simple example (which I suspect most of the business do) is that you do all work either manually yourself or on your laptop while advise them as a resource-rich service. Only when you truly can not handle the demand then you may 'scale up' and turn your business into 'real' business. And there are plenty of tricks like this (as legally as possible).
> Everytime you go to SO, it hits one of these 9 web servers and all data on SO sits on those 2 massive SQL servers. That's pretty amazing.
I don't find it amazing at all. Functionality-wise, StackOverflow is a very simple Web application. Moreover, SO's range of 300-500 requests per second is not a mind-blowing load. Even in 2014, a powerful enough single physical server (running a Java application) was able to handle 1M requests per second[1]. A bit later, in 2017, similar performance has been demonstrated on a single AWS EC2 instance, using Python (and a blazingly-fast HTTP-focused micro-framework Japronto), which is typically not considered a high-performance option for Web applications[2].
[1] https://www.techempower.com/blog/2014/03/04/one-million-http...
[2] https://www.freecodecamp.org/news/million-requests-per-secon...
I don't find it amazing at all. Functionality-wise, StackOverflow is a very simple Web application. Moreover, SO's range of 300-500 requests per second is not a mind-blowing load. Even in 2014, a powerful enough single physical server (running a Java application) was able to handle 1M requests per second[1]. A bit later, in 2017, similar performance has been demonstrated on a single AWS EC2 instance, using Python (and a blazingly-fast HTTP-focused micro-framework Japronto), which is typically not considered a high-performance option for Web applications[2].
[1] https://www.techempower.com/blog/2014/03/04/one-million-http...
[2] https://www.freecodecamp.org/news/million-requests-per-secon...
The amaziness is that the leadership allows it to be simple.
This is such a great competitive advantage.
Compare this to a leadership that thinks you absolutely must use Akamai for your 50 req/secs webserver. You end up with tons of complexity for no reason.
This is such a great competitive advantage.
Compare this to a leadership that thinks you absolutely must use Akamai for your 50 req/secs webserver. You end up with tons of complexity for no reason.
Fair enough. Though not too surprising still, considering the original leadership of the company, one of whom (Joel Spolsky) is still on the board of directors. Having said that, the board's 5:4 VC-to-non-VC ratio looks pretty scary to me. But this is a different story ...
SO is a bit more complicated than returning a single character in a response. You can achieve high throughput with just about anything these days if you aren't doing any "work" on the server. 300-500 reqs/second is impressive for a web site/application with real-world traffic.
Thing is 99% of companies could run like SO if their software would be like SO.
But if you are confronted with a very large 15+ year old monolith that requires multiple big instance machines to even handle medium load. Then you're not going to get this easily fixed.
It's very possible that you come to the conclusion that it is too complex to refactor for better vertical scaling. When your demand increases, then you simply buy another machine every now and then and spin up another instance of your monolith.
But if you are confronted with a very large 15+ year old monolith that requires multiple big instance machines to even handle medium load. Then you're not going to get this easily fixed.
It's very possible that you come to the conclusion that it is too complex to refactor for better vertical scaling. When your demand increases, then you simply buy another machine every now and then and spin up another instance of your monolith.
> if you are confronted with a very large 15+ year old monolith that requires multiple big instance machines to even handle medium load. Then you're not going to get this easily fixed
Last 15+ year old monolith I touched needed multiple machines to run because it was constrained by the database due to an insane homegrown ORM and poorly managed database schemas (and this is a common theme, I find.)
Tuning the SQL, rejigging things like session management, etc., would have made it go a lot quicker on a lot fewer machines but management were insistent that it had to be redone as node microservices under k8s.
Last 15+ year old monolith I touched needed multiple machines to run because it was constrained by the database due to an insane homegrown ORM and poorly managed database schemas (and this is a common theme, I find.)
Tuning the SQL, rejigging things like session management, etc., would have made it go a lot quicker on a lot fewer machines but management were insistent that it had to be redone as node microservices under k8s.
I totally agree with your main point and SO is kind of the perfect example. At the same time it is kind of the worst example because for one, to the best of my knowledge, their architecture is pretty much an outlier, and for another it is what it is for non-technical historical reasons.
As far as I remember they started that way because they were on a Microsoft stack and Microsofts licensing policies were (are?) pretty much prohibitive for scaling out. It is an interesting question if they would design their system the same way if they'd the opportunity to start from scratch.
As far as I remember they started that way because they were on a Microsoft stack and Microsofts licensing policies were (are?) pretty much prohibitive for scaling out. It is an interesting question if they would design their system the same way if they'd the opportunity to start from scratch.
Most people responding here are nitpicking on whether SO’s architecture is the one. I wasn’t trying to imply that at all.
I wanted to drive a point and SO is a good enough example to show that a massive company of the size of SO can run, so can your tiny app.
Don’t scale prematurely. A lot can be done by reasonable vertical scaling.
For one $120k/year Kubernetes infra engineer, you could pay for entire rack of beefy servers.
Obviously YMMV. Discussion about SO and licensing details are distracting.
I wanted to drive a point and SO is a good enough example to show that a massive company of the size of SO can run, so can your tiny app.
Don’t scale prematurely. A lot can be done by reasonable vertical scaling.
For one $120k/year Kubernetes infra engineer, you could pay for entire rack of beefy servers.
Obviously YMMV. Discussion about SO and licensing details are distracting.
Yes but Stackoverflow is a now mostly a graveyard of old closed questions, easily cached, I am only half joking.
Most startup ideas today are a lot more interactive, so a SO model with two DBs would probably not serve them well. Horizontal scaling is not only for ETL and I am uncertain in why you say that it needs many lawyers.
Related, LetsEncrypt recently published a blogpost about their database servers and how they scale vertically: https://letsencrypt.org/2021/01/21/next-gen-database-servers...
Genuine question, how is 9 web servers vertical scaling? And also, peak CPU usage of 12% means this is about 10x oversized for what is needed. Isn't it much better to only scale up when actually needed, mostly in terms of cost?
because they play in the major leagues where most teams have hundreds or thousands of servers, while they have those nine.
Yes, there is some horizontal scaling, but the sheer amount of vertical scaling here is still mind blowing.
I've run more servers in what were basically hobby projects compared to SO.
Yes, there is some horizontal scaling, but the sheer amount of vertical scaling here is still mind blowing.
I've run more servers in what were basically hobby projects compared to SO.
Probably 9 different geographical locations. Nothing to do with the actual load per server
Stack-overflow's use case has the benefit of being able to sit behind a Content Delivery Network (CDN) with a massive amount of infrastructure at the edge offloading much of the computational and database demands. This reduces the requirements of their systems dramatically. Given their experience in the segment, its plausible to expect they understand how to optimize their user-experience to balance out the hardware demands and costs as well.
Looks like there's also two Redis servers with half a TB of RAM sitting in between web and SQL. I'm sure that's a huge load off the SQL server.
https://meta.stackoverflow.com/a/306604/2489265
https://meta.stackoverflow.com/a/306604/2489265
I think I agree, but what do you mean exactly? Just keep getting beefier servers as opposed to serverless junk?
Not the OP, but yes, getting more powerful machines to run your program is what "vertical scaling" means (as opposed to running multiple copies of your program on similar-sized machines aka "horizontal scaling" ).
A ‘single’ big box with multiple terabytes of RAM can probably outperform many ‘horizontally scaled’ solutions. It all depend on the workload, but I feel that sometimes it’s more about being ’hip’ than being practical.
https://yourdatafitsinram.net/
https://yourdatafitsinram.net/
Might apply to 99 % of the companies but I doubt it applies to 99 % of the companies that HN readers work for.
Their database query to page request ratio is about 20:1. Seems like this should be lower.
Stack Overflow have unique constraints ( Microsoft licensing) which make vertical scaling a cheaper option, and IMHO that's rarely the case.
people keep fapping to this, but stackoverflow is served read only to most users, and probably heavy cached.
What is vertical scaling?
Use bigger machines instead of more machines.
There's always a limit on how big you can go, and a smaller limit on how big you should go, but eitherway it's pretty big. I wouldn't go past dual Intel Xeon, because 4P gets crazy expensive; I haven't been involved in systems work on Epyc, 1P might be a sensible limit, but maybe 2P makes sense for some uses.
There's always a limit on how big you can go, and a smaller limit on how big you should go, but eitherway it's pretty big. I wouldn't go past dual Intel Xeon, because 4P gets crazy expensive; I haven't been involved in systems work on Epyc, 1P might be a sensible limit, but maybe 2P makes sense for some uses.
If you have a single machine with 64 cores running 256 threads of your daemon that's considered vertical scaling? Odd definition
If multiple cores/threads shouldn't be considered vertical scaling, what should?
Overclocking a single-core processor from 2.8Ghz to 4.2Ghz can only take you so far after all...
Overclocking a single-core processor from 2.8Ghz to 4.2Ghz can only take you so far after all...
Yes, because it's still a single system image (one running OS with one process table).
'Scale up' vs 'scale out' are, to me, more intuitive terms for the same thing. Up/vertical are aligned, I guess
Usually it involves having an insane amount of ram and keeping the entire DB in ram.
I have found this to be a helpful resource when it comes to scaling in general.
https://alexpareto.com/scalability/systems/2020/02/03/scalin...
https://alexpareto.com/scalability/systems/2020/02/03/scalin...
Get a more powerful single machine (in contrast to multiple machines).
However I wonder if multisockets Xeons count as vertical or horizontal. I never understood how programmable those machines are..
[deleted]
It might apply to 99% who have specific requirements, but the vast majority of internet companies need more. Deployments, N+1 redundancy, HA etc... are all valuable, even if some resources are going to waste.
> Deployments, N+1 redundancy, HA etc
None of those things are mutually exclusive with vertical scaling?
Having two identical servers for redundancy is doesn't mean you are scaling horizontally (assuming each can handle the load individually, nothing better than discovering that that assumption was incorrect in an outage).
None of those things are mutually exclusive with vertical scaling?
Having two identical servers for redundancy is doesn't mean you are scaling horizontally (assuming each can handle the load individually, nothing better than discovering that that assumption was incorrect in an outage).
Indeed, all of those things existed before docker.
The author is not lying. I've been learning docker and it is certainly nice to pull and docker-compose up for local dev, but there is a lot to learn when you factor in orchestrators. And when I mean learn, I mean actually learning the nuts and bolts of a Dockerfile and not copy/pasting shit from the internet. While it's all helpful, its certainly not needed for the project we have at work, nor the "microservices" that our lead thinks we need even though they aren't even microservices. All we've done is split our application into seperate mini applications that don't talk to eachother at all, essentially 4 web services. Great! Sigh.
So why am I learning all this instead of just programming? Programming is what I really want to do all day, just writing code and thinking through software architecture. Because the industry tells me if I don't learn this stuff I will be decommissioned. Fun stuff.
So why am I learning all this instead of just programming? Programming is what I really want to do all day, just writing code and thinking through software architecture. Because the industry tells me if I don't learn this stuff I will be decommissioned. Fun stuff.
You're learning this stuff because you're an engineer not a computer scientist. Deploying to prod is the goal, not writing code.
>seperate mini applications that don't talk to eachother at all, essentially 4 web services.
I mean, that sounds pretty good. If you can do that why couple them? "Microservices" is just SOA without any debate over how small a service can be.
>seperate mini applications that don't talk to eachother at all, essentially 4 web services.
I mean, that sounds pretty good. If you can do that why couple them? "Microservices" is just SOA without any debate over how small a service can be.
Whether decoupling into microservices or making them part of a monolith is a good idea or not, to me, had a lot more to do with the teams and company organization behind these services than the product itself.
So I'd say without knowing how big, and how many teams are dealing with this we cannot say one approach is better than the other.
I've been in companies where several hundreds of engineers where working on a "core" monolith and it was a real pain, almost impossible to track where problems where coming from, some times deploys blocked for weeks due to some team having problems with their tests or just spending months to agree how to do something.
I've also been at companies where being about 10 devs the "architect" went totally crazy into microservices, microfraneworks, cqrs, event sourcing, etc etc... And it was a total mess where each developer had to deal with 10 services and keep it's dependencies up to date and coordinate deploys of 3 things at the same time and nobody knew where and why things were failing.
So, as always, right tool for the job.
What I've seen works the best, is to adapt the services you run to the teams structure you have.
So I'd say without knowing how big, and how many teams are dealing with this we cannot say one approach is better than the other.
I've been in companies where several hundreds of engineers where working on a "core" monolith and it was a real pain, almost impossible to track where problems where coming from, some times deploys blocked for weeks due to some team having problems with their tests or just spending months to agree how to do something.
I've also been at companies where being about 10 devs the "architect" went totally crazy into microservices, microfraneworks, cqrs, event sourcing, etc etc... And it was a total mess where each developer had to deal with 10 services and keep it's dependencies up to date and coordinate deploys of 3 things at the same time and nobody knew where and why things were failing.
So, as always, right tool for the job.
What I've seen works the best, is to adapt the services you run to the teams structure you have.
> You're learning this stuff because you're an engineer not a computer scientist. Deploying to prod is the goal, not writing code.
Writing code is not the goal of a computer scientist.
Writing code is not the goal of a computer scientist.
Because there is a bunch of shared code behavior and data between the services and his idea was "just copy and paste". We devised a library that gets shared as a dependency, but managing that dependency is PITA and not everything can go into the shared library still. Which means we still copy/paste some code, which is okay with this bozo somehow. I can stomach splitting things into microservices that don't actually talk to each other, though I don't see the need day one, but why not run it as a monorepo and split it out in the CD pipeline so I don't have to open four PRs for one fucking task?
Because the goof couldn't figure it out and just gave up and said here you go. This is before I knew the guy was completely useless. I had to do a whole presentation and get the favor of every single developer just for him to succumb to FACTS about moving to a monorepo that then splits the code out to his beloved CV fuel on build. Trust me, there is a lot more wrong here if you are getting any of this. As for team size we are small. Six warm bodies, three actually moving.
Personally I say go monolith (for most small/medium projects) but design it with an eye towards splitting it into services down the line, that is unless something is smacking you in the face to go all in on micro. For monolithic that means a logical separation today, that can easily be split out in a single sprint tomorrow. That means less pain and more features today and a path towards some dudes wet dream in the future.
Yeah I'm salty.
Because the goof couldn't figure it out and just gave up and said here you go. This is before I knew the guy was completely useless. I had to do a whole presentation and get the favor of every single developer just for him to succumb to FACTS about moving to a monorepo that then splits the code out to his beloved CV fuel on build. Trust me, there is a lot more wrong here if you are getting any of this. As for team size we are small. Six warm bodies, three actually moving.
Personally I say go monolith (for most small/medium projects) but design it with an eye towards splitting it into services down the line, that is unless something is smacking you in the face to go all in on micro. For monolithic that means a logical separation today, that can easily be split out in a single sprint tomorrow. That means less pain and more features today and a path towards some dudes wet dream in the future.
Yeah I'm salty.
> I mean, that sounds pretty good. If you can do that why couple them?
Because what is the point in separating each one into their own deployed service and having to deal with network issues when they could just be services inside a monolith?
Because what is the point in separating each one into their own deployed service and having to deal with network issues when they could just be services inside a monolith?
OP just said they don't talk at all. They sound completely decoupled already. You could put them in the same app and share routes if you want and just separate by package or compilation unit if you really want to. The hard part is already done though.
They could be managed and deployed by completely different teams with no overhead now. It really depends what we're tuning for.
They could be managed and deployed by completely different teams with no overhead now. It really depends what we're tuning for.
Scaling is another factor that comes to mind, also resilience.
If one part in the monolith goes haywire so will the entire application. If you can decouple and split the software into 4 applications with their own concerns, at least you have 3 applications left running (assuming they are decoupled).
If app 1 wants 5000% more CPU than app 2, maybe you can have different instance types running and save costs/resources.
If one part in the monolith goes haywire so will the entire application. If you can decouple and split the software into 4 applications with their own concerns, at least you have 3 applications left running (assuming they are decoupled).
If app 1 wants 5000% more CPU than app 2, maybe you can have different instance types running and save costs/resources.
A good reason no doubt and if that comes to bear sure. But at least code in a monorepo and split to microservices in your pipeline. Cake and eat.
Read my reply further up, they have a lot in common. We had to build a shared library that goes in as a dependency and even with that we have to still copy/paste code between services. Because a data structure change in one must often be reflected in another HELLO multiple PRs for one task.
Developing with docker is not necessary a micro service, it's just a way of packaging, distributing and deploying your application in a clean way. And docker is not a virtual machine, there's not much overhead, you don't need kubernetes if it's just a simple app, but you can just take advantage of managed service like ECS, you get auto scaling right away, and you don't have manage your node and deal with the stupid thing like systemD
Yup, I see docker (and K8s) as another Websphere.
> you get auto scaling right away
Every time I see someone say something like this, or better yet use the word "magic", what I hear is that they don't understand how or why their system does the things it does. Nothing is free; nothing is magic; nothing "just works". You understand it or you don't, and nothing has contributed more to the plague of developers thinking they understand it when they don't than cloud-based containers (and the absurd associated costs!).
Every time I see someone say something like this, or better yet use the word "magic", what I hear is that they don't understand how or why their system does the things it does. Nothing is free; nothing is magic; nothing "just works". You understand it or you don't, and nothing has contributed more to the plague of developers thinking they understand it when they don't than cloud-based containers (and the absurd associated costs!).
> Every time I see someone say something like this, or better yet use the word "magic", what I hear is that they don't understand how or why their system does the things it does.
There is nothing magic about configuring a deployment to autoscale. You set resource limits, you configure your deployment to scale up if an upper limit is reached and you didn't maxed out, and you configure your deployment to scale down if a lower limit is reached and you didn't min out. What do you find hard to understand?
> Nothing is free; nothing is magic; nothing "just works". You understand it or you don't,
You're the only one claiming it's magic.
The rest of the world doesn't seem to have a problem after having read a couple of pages into the tutorial on how to configure autoscale on whatever service/cloud provider.
> and nothing has contributed more to the plague of developers thinking they understand it when they don't than cloud-based containers (and the absurd associated costs!).
You're the only one posting a baseless assertion that other developers somehow don't understand autoscaling, as if everyone around you struggles with it.
There is nothing magic about configuring a deployment to autoscale. You set resource limits, you configure your deployment to scale up if an upper limit is reached and you didn't maxed out, and you configure your deployment to scale down if a lower limit is reached and you didn't min out. What do you find hard to understand?
> Nothing is free; nothing is magic; nothing "just works". You understand it or you don't,
You're the only one claiming it's magic.
The rest of the world doesn't seem to have a problem after having read a couple of pages into the tutorial on how to configure autoscale on whatever service/cloud provider.
> and nothing has contributed more to the plague of developers thinking they understand it when they don't than cloud-based containers (and the absurd associated costs!).
You're the only one posting a baseless assertion that other developers somehow don't understand autoscaling, as if everyone around you struggles with it.
Having separate services does add additional overhead and maintenance, but it does provide the benefit of 1) Reducing the blast radius if issues occur, allowing for a degraded service instead of being totally down, and 2) Better scaling/ optimizations. For example, one service could need to support more TPS, or need more memory, CPU, etc.
if they don't talk to each other, then you can run multiple instances of them without paying the price of intercommunications latency. It's a dream for scaling. In this example their teamlead is right.
Because it looks good on a resume.
Isn't that why the DevOps role was invented, so us programmers don't have to fanny around with docker.
Devops is a practice not a role. Essentially it’s bringing good software engineering practice to operations and eliminating the silos. In practice, though, this does usually mean programmers fannying around with docker.
Some organisations just rebranded the ops team to the devops team, but that’s kinda missing the point.
I'm actually perplexed why the people in our ops team have the title DevOps Engineer, when they don't do any dev work, just handle all aws related stuff.
I asked one of them this question, but couldn't get any answer that satisfied my curiousity.
EDIT: For what it's worth I don't view myself as a software developer or operations or systems administrator or frontend or backend or fullstack. I like to think of myself as a Problem Solver. It just so happens that I'm currently paid to solve software engineering problems.
I want to know everything from Dockerfiles to bash scripting to assembly to functional programming to DDD, etc.
I asked one of them this question, but couldn't get any answer that satisfied my curiousity.
EDIT: For what it's worth I don't view myself as a software developer or operations or systems administrator or frontend or backend or fullstack. I like to think of myself as a Problem Solver. It just so happens that I'm currently paid to solve software engineering problems.
I want to know everything from Dockerfiles to bash scripting to assembly to functional programming to DDD, etc.
I have asked this question myself, and after failing to get an appropriate answer, I started using "DevOps Administrator" instead of "Engineer" in my email signature. It feels more appropriate since I definitely do not write production code.
In my experience, developers then still have to implement a lot of behaviour to deal with docker's limitations around networking. (in my case it was trying to connect multiple BEAMs in a docker network)
And here we come full circle. Let the s̶y̶s̶a̶d̶m̶i̶n̶ devops handle it.
For me, docker is just a superpower that let's me build larger, more complex applications. That's why it's worth learning. It raises the ceiling of what I'm able to create. Creating a site with multiple services that integrate with each other is complicated. You could do it from scratch, but it would take so long that you would likely never be able to manage the actual complexity you care about--the complexity you want to architect and program. You'd be too busy programming solutions to the problems that docker already solves.
> You could do it from scratch, but it would take so long that you would likely never be able to manage the actual complexity you care about
Except plenty of people have and continue to manage? Do we think that _most_ people are using Docker now?
This is how we do it:
We use baremetal servers. We use systemd, including the systemd sandboxing capabilities. We have deploy scripts (almost all bash, and yes, they are largely idempotent). A host could be deployed with any number of: cockroachdb, openresty, monetdb, rabbitmq or postgresql. They're all running vector (log ingestion). And they can be running N apps. Each host also get public keys for encrypting on-site, credentials for uploading backups to Google Cloud, systemd-timers for tasks like running lynis and rkhunter. Everything is connected via wireguard (over a physical private network).
Our apps (mostly go and elixir) are built on a self-hosted gitlab and we deploy like normal. Each project can pick what it wants, e.g. auto deploy dev branch to dev, auto deploy master to stage, manually-triggered to prod.
We run separate tool servers for our internal stuff, gitlab, elastic search, kibana, prometheus, vault and grafana.
We have some clustering/mesh networking which we "discover" with an internal DNS. But everything is pretty static/fixed. We can survive downtime because either something is HA and stateless (e.g. we keep N monetdb "reporting" servers in sync from the OLTP->OLAP process), or stateful and part of a cluster (e.g., cockroachdb).
Here's the kicker: we have 0 devops. We don't spend a huge deal of time on this. We used to have devops and K8 and everything was worse. It was _much_ more expensive, everything took _a lot_ longer to get done, it all ran slower, it was much buggier, and it was less secure (because we aren't sharing a host).
I feel like we could build almost anything this way. I don't see what complexity we'd run into that Docker would help with. I think we'd run into management issues...applying patches to hundreds of servers, reviewing hardening and access logs, etc. But that's infrastructure complexity, not application complexity.
We use docker in apps that require a complex setup for end-to-end testing.
Except plenty of people have and continue to manage? Do we think that _most_ people are using Docker now?
This is how we do it:
We use baremetal servers. We use systemd, including the systemd sandboxing capabilities. We have deploy scripts (almost all bash, and yes, they are largely idempotent). A host could be deployed with any number of: cockroachdb, openresty, monetdb, rabbitmq or postgresql. They're all running vector (log ingestion). And they can be running N apps. Each host also get public keys for encrypting on-site, credentials for uploading backups to Google Cloud, systemd-timers for tasks like running lynis and rkhunter. Everything is connected via wireguard (over a physical private network).
Our apps (mostly go and elixir) are built on a self-hosted gitlab and we deploy like normal. Each project can pick what it wants, e.g. auto deploy dev branch to dev, auto deploy master to stage, manually-triggered to prod.
We run separate tool servers for our internal stuff, gitlab, elastic search, kibana, prometheus, vault and grafana.
We have some clustering/mesh networking which we "discover" with an internal DNS. But everything is pretty static/fixed. We can survive downtime because either something is HA and stateless (e.g. we keep N monetdb "reporting" servers in sync from the OLTP->OLAP process), or stateful and part of a cluster (e.g., cockroachdb).
Here's the kicker: we have 0 devops. We don't spend a huge deal of time on this. We used to have devops and K8 and everything was worse. It was _much_ more expensive, everything took _a lot_ longer to get done, it all ran slower, it was much buggier, and it was less secure (because we aren't sharing a host).
I feel like we could build almost anything this way. I don't see what complexity we'd run into that Docker would help with. I think we'd run into management issues...applying patches to hundreds of servers, reviewing hardening and access logs, etc. But that's infrastructure complexity, not application complexity.
We use docker in apps that require a complex setup for end-to-end testing.
What you have sounds great, and sounds about the same as running containers with orchestration, just via SystemD cgroups instead of container cgroups, and bash and humans instead of a real orchestrator. You lose a lot of features ( idempotent and self-contained ( so zero risk of dependency clashing), cluster-wide auto-healing, load distribution and autoscaling, centralised management) and ease of use of existing solutions ( want to run a CockroachDB cluster? There's probably a k8s operator for that; want automatic DNS? Consul from Nomad, k8s already do that), but you understand it. There could be very serious downsides around maintenance of your bash scripts, and onboarding new people.
Personally, I don't want to think about all this complexity. I just build simple web applications. I want a box in my basement that has everything production has - the whole thing, all the web services running all the time.
It ought to be possible in my mind. After all, there is really only one user: me. I just want to be able to automatically get the latest code that reservations team or order management system team or what have you deploys to production.
Why do I need to connect to VPN just to work on a stupid web application? None of the code we write has any special sauce in it and we can have fake data...
It ought to be possible in my mind. After all, there is really only one user: me. I just want to be able to automatically get the latest code that reservations team or order management system team or what have you deploys to production.
Why do I need to connect to VPN just to work on a stupid web application? None of the code we write has any special sauce in it and we can have fake data...
> largely idempotent
Got to say, that "largely" scares me when discussing deployments.
Got to say, that "largely" scares me when discussing deployments.
I love docker as a home user as well. Sometimes when I'm bored I'll build an IM bot. Docker makes hosting it super simple. I just start from the ruby docker image, copy my script in and then dump it on gitlab which will build and host the container which lets me pull it from my server.
I can then copy a simple systemd config to start it on boot and restart it if it fails. This is all much simpler than managing multiple versions of ruby installed locally. Not to mention the security benefits of keeping it in a container. Perhaps this is all so easy and convenient for me because I learned docker for pro use and now its just mentally free for casual use.
I can then copy a simple systemd config to start it on boot and restart it if it fails. This is all much simpler than managing multiple versions of ruby installed locally. Not to mention the security benefits of keeping it in a container. Perhaps this is all so easy and convenient for me because I learned docker for pro use and now its just mentally free for casual use.
Docker has autorestart on failure. I think it also autorestarts after reboot.
Yeah I have seen this but for some reason it just wasn't working for me. I think that maybe the docker service just wasn't being started at boot but once I set it up with systemd it all just works now.
I find it painfully reliable. I forget about it and then weeks later I find the image when scrounging about trying to work out where my RAM went.
I started using Docker heavily about 3 years into my current project and that was the right time. There's real overhead to getting things to work on compared to a local dev environment, but there are major benefits, especially once you actually do need to run the same thing on multiple machines. It's a pretty classic example of something with a high constant factor but better scaling properties.
Unfortunately I also had to waste my time learning containers when all I wanted was Heroku without the insane prices for addon services. For delivering just apps I think Cloud Native Buildpacks solve that: https://buildpacks.io/
What does "delivering just apps" means?
And what's buildpacks? Seems like a more specialized docker.
And what's buildpacks? Seems like a more specialized docker.
Buildpacks predate Docker. They're standardized build scripts for popular app runtimes. Heroku created the concept and applied it for their PaaS years before Docker was a thing. They're not a competitor to Docker containers. They are perhaps a competitor to Dockerfiles.
They detect the kind of app and then basically do everything required to build and run that app from pushing a repository. So it'll detect that it's a NodeJS application because there's a package.json file in the root and essentially: install node modules, run the build script and then do npm run start to run it.
All you need to know how to do is: supply a build + start script and push your repository and now it's built and running in the Cloud to scale from 0 to unlimited.
It's basically Heroku but buildpacks are a standardised way to do it on any Cloud.
All you need to know how to do is: supply a build + start script and push your repository and now it's built and running in the Cloud to scale from 0 to unlimited.
It's basically Heroku but buildpacks are a standardised way to do it on any Cloud.
Render.com is like Heroku but cheaper and handles static sites as well as distributed applications (nodes can talk to each other).
Have you tried Dokku? Very similar to Heroku. I run it on a DO server without any issues.
> ... the "microservices" that our lead thinks we need even though they aren't even microservices. All we've done is split our application into seperate mini applications that don't talk to eachother at all,
Replacing in-app API/libary calls with RPC is an micro-service anti-pattern (1). If the services don't need to communicate you have probably made the correct split - or you could think backwards - why would these services that are independent and don't talk to each need to be merged into a monolith ?
1) There are of course exceptions, there could be a good idea to separate things for performance, eg. create worker processes. Which works if they are like pure functions.
Replacing in-app API/libary calls with RPC is an micro-service anti-pattern (1). If the services don't need to communicate you have probably made the correct split - or you could think backwards - why would these services that are independent and don't talk to each need to be merged into a monolith ?
1) There are of course exceptions, there could be a good idea to separate things for performance, eg. create worker processes. Which works if they are like pure functions.
As my reply far up this chain says, the services have a ton in common. So much so that the idea was "just copy and paste" code between. Still sound great? I forced his hand on a shared library that houses lots of the code all services need, but not everything can (or should) go in there.
Multiple PRs just to complete one task sometimes and we're talking a small 4-6 hour task. If the services were truly independent this wouldn't be needed and the approach wouldn't be a poor developer experience and infrastructure headache. But a change in one often requires a change in another. We don't have a monorepo, because he was too much of a doofus to figure it out and gave up instead of asking for help.
80% of the problems are because the guy doesn't know what he's doing, doesn't know architecture and just said we're doing this. Which means I have to go in and fix his poorly built Dockerfiles that are an expose on what not to do. We're now adding API Gateway. Was this explained as to why we need it? No. Did I ask yes? But I simply get "I've already explained it" Cool. I explained to you why running an update on packages in your build instead of installing from a lock file is some of the dumbest shit I've ever seen yet I just had to go in and clean up your dockerfile again. You want untested packages going into stage / prod? My lead is your guy. I'm sure in a few weeks he'll come to me with "But API Gateway doesn't work and I don't have time can you fix it".
Fuck this dude. I just want to write clean code and not fuck with his mistakes. Did I mention we run completely different Dockerfiles between environments (local vs stage/prod). Like, not even the same O.S (ubuntu vs alpine), web server (apache vs nginx) etc... Getting the picture of whats its like to deal with his mistakes day in and day out and slowly fix them.
Multiple PRs just to complete one task sometimes and we're talking a small 4-6 hour task. If the services were truly independent this wouldn't be needed and the approach wouldn't be a poor developer experience and infrastructure headache. But a change in one often requires a change in another. We don't have a monorepo, because he was too much of a doofus to figure it out and gave up instead of asking for help.
80% of the problems are because the guy doesn't know what he's doing, doesn't know architecture and just said we're doing this. Which means I have to go in and fix his poorly built Dockerfiles that are an expose on what not to do. We're now adding API Gateway. Was this explained as to why we need it? No. Did I ask yes? But I simply get "I've already explained it" Cool. I explained to you why running an update on packages in your build instead of installing from a lock file is some of the dumbest shit I've ever seen yet I just had to go in and clean up your dockerfile again. You want untested packages going into stage / prod? My lead is your guy. I'm sure in a few weeks he'll come to me with "But API Gateway doesn't work and I don't have time can you fix it".
Fuck this dude. I just want to write clean code and not fuck with his mistakes. Did I mention we run completely different Dockerfiles between environments (local vs stage/prod). Like, not even the same O.S (ubuntu vs alpine), web server (apache vs nginx) etc... Getting the picture of whats its like to deal with his mistakes day in and day out and slowly fix them.
It's super simple and it works for you - yeay! great!
If it will continue to work for you - double yeay!
However, as soon as I read, I saw a lot of red flags:
- Do you really want to copy from a development computer to your production? No staging at all? ("go test" doesn't mean that you have 0 bugs)
- Are you really sure that everything works exactly the same on different versions of GoLang? (Hey, a new guy in your company just installed unreleased Go 1.17, build on his notebook and pushed to production)?
- That VM with systemd died at 1am. No customers for you until 7am (when you wake up)
BTW. I am not saying that you should do Docker or CICD. The thing which I am saying that when you cut out from your process too much, you are increasing risks. (As an example, you didn't remove unit tests part. Based on "Anything that doesn’t directly serve that goal is a complication" you probably should have. However, you decided that it would be way too much risk)
However, as soon as I read, I saw a lot of red flags:
- Do you really want to copy from a development computer to your production? No staging at all? ("go test" doesn't mean that you have 0 bugs)
- Are you really sure that everything works exactly the same on different versions of GoLang? (Hey, a new guy in your company just installed unreleased Go 1.17, build on his notebook and pushed to production)?
- That VM with systemd died at 1am. No customers for you until 7am (when you wake up)
BTW. I am not saying that you should do Docker or CICD. The thing which I am saying that when you cut out from your process too much, you are increasing risks. (As an example, you didn't remove unit tests part. Based on "Anything that doesn’t directly serve that goal is a complication" you probably should have. However, you decided that it would be way too much risk)
Exactly my reasons: I don‘t use Docker because it‘s great that i can (could?) scale the universe.
In my case, I simply use Docker because it is so easy to set up n HTTP services listening on port 80/442 on the same server and then put a reverse proxy (traefik) with Let‘s Encrypt in front of it. I don‘t need to worry about port conflicts. I don‘t need to worry about running Nginx and Apache on the same host. I don‘t need to worry about multiple versions of Go/PHP/Dotnet/[insert lang here].
Still, I can‘t scale (single machine) but I don‘t need to. I don‘t have a failover, because I don‘t need one. But I have so much simpler management of the dozen services I run on the host. And that‘s worth it IMHO.
I think it‘s always about the right tool for the job. And I think if the OP does work with an automated script and scp, there‘s nothing wrong with that. Because that also adds reproducability to the pipeline and that‘s just such an important point. As long as nobody ssh‘s into prod and modifies some file by hand.
In my case, I simply use Docker because it is so easy to set up n HTTP services listening on port 80/442 on the same server and then put a reverse proxy (traefik) with Let‘s Encrypt in front of it. I don‘t need to worry about port conflicts. I don‘t need to worry about running Nginx and Apache on the same host. I don‘t need to worry about multiple versions of Go/PHP/Dotnet/[insert lang here].
Still, I can‘t scale (single machine) but I don‘t need to. I don‘t have a failover, because I don‘t need one. But I have so much simpler management of the dozen services I run on the host. And that‘s worth it IMHO.
I think it‘s always about the right tool for the job. And I think if the OP does work with an automated script and scp, there‘s nothing wrong with that. Because that also adds reproducability to the pipeline and that‘s just such an important point. As long as nobody ssh‘s into prod and modifies some file by hand.
100%. For the startup we're starting/working on now, we're running microk8s on bare metal (dedicated servers).
What you describe is a big reason for it. Once you have k8s set up, services can be very easily deployed with auto TLS, and basic auth or oauth2 is really simple as well.
So we're big believers in vertical scaling, but still use k8s (microk8s or k3s) for these kinds of benefits. An additional benefit of this is that it makes scaling/transitioning to a bigger/managed k8s easy down the road.
It might sound like overkill, but it takes about 10 minutes to set up microk8s on a ubuntu lts server. It comes with nginx ingress. Install cert manager (another 5 mins) and you don't even need traefik these days. All configs are kept in Git.
What you describe is a big reason for it. Once you have k8s set up, services can be very easily deployed with auto TLS, and basic auth or oauth2 is really simple as well.
So we're big believers in vertical scaling, but still use k8s (microk8s or k3s) for these kinds of benefits. An additional benefit of this is that it makes scaling/transitioning to a bigger/managed k8s easy down the road.
It might sound like overkill, but it takes about 10 minutes to set up microk8s on a ubuntu lts server. It comes with nginx ingress. Install cert manager (another 5 mins) and you don't even need traefik these days. All configs are kept in Git.
> it takes about 10 minutes
After you've spent weeks / months reading blogs, installing dependencies, tweaking parameters, playing around deployments and figuring out new names for the same old stuff, debugging that extra whitespace in the yaml file only to figure out oww, you could use helm charts for deployments which is eerily similar to how jsp looked like 15 years ago and everything that was wrong with templating deployments than scripting it.
Then it takes about 4 mins.
And now you get to do brown bag sessions with your baffled team members! Yay!
But only till kubernetes evicts your pods for being too resource "hungry". Gotta keep prod up folks. Better grab a coffee and kubectl ("kube-cuttle" is it?) into prod to check why the pods have restarted 127 times.
After you've spent weeks / months reading blogs, installing dependencies, tweaking parameters, playing around deployments and figuring out new names for the same old stuff, debugging that extra whitespace in the yaml file only to figure out oww, you could use helm charts for deployments which is eerily similar to how jsp looked like 15 years ago and everything that was wrong with templating deployments than scripting it.
Then it takes about 4 mins.
And now you get to do brown bag sessions with your baffled team members! Yay!
But only till kubernetes evicts your pods for being too resource "hungry". Gotta keep prod up folks. Better grab a coffee and kubectl ("kube-cuttle" is it?) into prod to check why the pods have restarted 127 times.
All these “it takes 10 minutes” should really be “it takes 10 minutes plus several hours/days/weeks actually learning how to run and maintain all this stuff”.
Precisely. Some weeks reading, then 10 minutes doing (well some hours or a day doing, more likely).
And then half a day downtime every now and then maybe? Until one knows enough about the new stack
And then half a day downtime every now and then maybe? Until one knows enough about the new stack
I would even go to a lower level. I use Docker swarm and Docker compose. It is reliable, simple and very effective.
Do you have any opinion on microk8s versus k3s?
> Do you really want to copy from a development computer to your production? ...
> Are you really sure that everything works exactly the same on different versions of GoLang? ...
He mentions he does have a build server which runs a 10 line shell script to download code and build the binary.
Builds happen on that server, and I assume it handles deploying the compiled binary (and systemd script?) to the target as well.
The build server would also have a "blessed" golang version. New guy code that uses new not-yet-blessed features would not compile.
> That VM with systemd died at 1am...
Your docker host died. All your containers die along with it. Docker alone cannot solve this category of issues anyway.
> Are you really sure that everything works exactly the same on different versions of GoLang? ...
He mentions he does have a build server which runs a 10 line shell script to download code and build the binary.
Builds happen on that server, and I assume it handles deploying the compiled binary (and systemd script?) to the target as well.
The build server would also have a "blessed" golang version. New guy code that uses new not-yet-blessed features would not compile.
> That VM with systemd died at 1am...
Your docker host died. All your containers die along with it. Docker alone cannot solve this category of issues anyway.
But you add k8s now, so that three hosts can orchestrate three other hosts. Success!
The infrastructure layer is expanding, to meet the needs of the expanding infrastructure layer
Devops is the new self-justifying expanding bureaucracy of the tech world.
k8s isn't the only option, Nomad is a much better fit here.
Now you have 3 problems.
It's not solving "your system went down", it's adding more layers of system that can go down.
It's not solving "your system went down", it's adding more layers of system that can go down.
Except now your system can handle the inevitable server going down without taking the entire site offline. It DOES solve the problem of a single host failure causing an outage. Yes, there are other types of outages you can have, but it certainly does reduce the occurrence of outages significantly.
Are you really trying to suggest that people can't use Kubernetes to increase their reliability?
Are you really trying to suggest that people can't use Kubernetes to increase their reliability?
Yeah, I guess I am. It's adding whole layers of complexity and configuration to the system. I understand that those layers of complexity and configuration are designed to make the system more resilient, but it depends on everyone getting everything right all the time. The "screw-up surface" is huge.
Ever seen a large system that has it's own job server and scripts for orchestration/deployment? Application code that checks the status of it's peers and runtime env to determine what should run? All glued together with decades old perl and bash with no documentation.
I'll take "more configuration in yaml" over that.
I'll take "more configuration in yaml" over that.
that's not a 1:1 comparison though.
Leave your nice clean K8s deployment paradise to cruft up for decades, and will it be any better? I doubt it - there'll be old Dockerfiles and weird bits of yaml that shouldn't work but do, and upgrading a version of anything will break random things.
So yes, I think I would prefer the decades of crufty perl and bash to decades of crufty outdated yaml. At least the bash scripts have a hope of doing what they say they do, and are likely to still execute as intended.
Leave your nice clean K8s deployment paradise to cruft up for decades, and will it be any better? I doubt it - there'll be old Dockerfiles and weird bits of yaml that shouldn't work but do, and upgrading a version of anything will break random things.
So yes, I think I would prefer the decades of crufty perl and bash to decades of crufty outdated yaml. At least the bash scripts have a hope of doing what they say they do, and are likely to still execute as intended.
Hum... No, Kubernetes is not a HA solution.
One can certainly create an HA cluster over some infrastructure set up by kubernetes, just as well as one can take a bunch of physical servers, set them up by hand, and create an HA cluster with them. K8s isn't adding anything to the availability.
One can certainly create an HA cluster over some infrastructure set up by kubernetes, just as well as one can take a bunch of physical servers, set them up by hand, and create an HA cluster with them. K8s isn't adding anything to the availability.
> Docker alone cannot solve this category of issues anyway.
Docker does comes with an orchestrator out of the box, it's called Docker Swarm. You may not use it, but it's there and it's up to you to use it or not. It's extremely simple to setup, a single command on the manager and another one on the worker. It support healthcheck, replication, etc.... all super simples to setup too.
Sure doing all theses will takes, what 30 minutes? Instead of the 5 he took for his deployment, but it does solve that issue, natively, out of the box.
Oh and my docker image always have the "blessed" [insert environment here] version, thus everyone always use it while testing locally. If you need to update it, anyone can do it easily, without any knowledge of the build server environment, nor any special access to it.
Docker does comes with an orchestrator out of the box, it's called Docker Swarm. You may not use it, but it's there and it's up to you to use it or not. It's extremely simple to setup, a single command on the manager and another one on the worker. It support healthcheck, replication, etc.... all super simples to setup too.
Sure doing all theses will takes, what 30 minutes? Instead of the 5 he took for his deployment, but it does solve that issue, natively, out of the box.
Oh and my docker image always have the "blessed" [insert environment here] version, thus everyone always use it while testing locally. If you need to update it, anyone can do it easily, without any knowledge of the build server environment, nor any special access to it.
- Staging: THe world ran well with people pushing PHP files onto live environments and it will continue to run well long after Docker gets replaced with something else.
- Versioning: It's pretty easy to ensure you have same versions on the same platform.
- Systemd: None of this means he does not have Pagerduty or similar setup.
Why do I say all of this? Because i ran really good businesses with similar architecture to his back in the day. Sure, I run docker now, but sometimes we tend to overcomplicate things.
If you have one app, and one server. There is No good reason to run a layer of Docker on it. None.
Elixir, Ruby, PHP, Node -- if your business has a monolith and can run on one server, guaranteed there is less to worry about when you remove Docker.
- Versioning: It's pretty easy to ensure you have same versions on the same platform.
- Systemd: None of this means he does not have Pagerduty or similar setup.
Why do I say all of this? Because i ran really good businesses with similar architecture to his back in the day. Sure, I run docker now, but sometimes we tend to overcomplicate things.
If you have one app, and one server. There is No good reason to run a layer of Docker on it. None.
Elixir, Ruby, PHP, Node -- if your business has a monolith and can run on one server, guaranteed there is less to worry about when you remove Docker.
> THe world ran well with people pushing PHP files onto live environments
No, it didn't. The world didn't fall apart, but it absolutely burned out lots of people who had to deal with this irresponsible way of doing things.
The way we do things is much, much better now. It is more complex, but that can be worth it. Don't romanticize a past that was absolute hell for a lot of people.
Source: inherited and maintained many of these dumpster fires.
No, it didn't. The world didn't fall apart, but it absolutely burned out lots of people who had to deal with this irresponsible way of doing things.
The way we do things is much, much better now. It is more complex, but that can be worth it. Don't romanticize a past that was absolute hell for a lot of people.
Source: inherited and maintained many of these dumpster fires.
> Source: inherited and maintained many of these dumpster fires.
And you imagine whoever inherits what's become of your current projects in 10 years is going to be happy? Shrug
And you imagine whoever inherits what's become of your current projects in 10 years is going to be happy? Shrug
No, but they will have some combination of declarative infrastructure, build scripts with error messages, and Docker images as a starting point.
I still maintain some of my 10-year-old code, by the way. Once I got it to build and deploy with modern tools, it has been much, much easier to keep it updated with patches and the latest server OS.
I still maintain some of my 10-year-old code, by the way. Once I got it to build and deploy with modern tools, it has been much, much easier to keep it updated with patches and the latest server OS.
>Elixir, Ruby, PHP, Node -- if your business has a monolith and can run on one server, guaranteed there is less to worry about when you remove Docker.
For Ruby at least you run into the problem of keeping all the development environments the same. It's not insurmountable by any means but it's a constant nagging annoyance. Especially so once we start talking about working on multiple projects that may be using different versions of Ruby Postgres etc. Being able to do a docker-compose up and have the exact same environment as production is huge.
For Ruby at least you run into the problem of keeping all the development environments the same. It's not insurmountable by any means but it's a constant nagging annoyance. Especially so once we start talking about working on multiple projects that may be using different versions of Ruby Postgres etc. Being able to do a docker-compose up and have the exact same environment as production is huge.
Docker has nothing to do with any of that though.
There artifact is a single statically linked binary/executable, not a docker container. They can build that binary once and pass it along a deployment pipeline i.e dev, test, prod changing config parameters for each environment but the exact same code running.
Systemd just like the various container runtimes supports auto restarts + logging. You can have the same alerting tools hang off your logs etc etc.
The fact they are not using Docker does not mean they can't have a proper build/deployment pipeline. The fact they are dealing with a single static executable makes building a pipeline and robust deployment far simpler as they have far fewer moving pieces.
If the author was deploying python, javascript, ruby apps where you don't get a static executable artifact or fat jar with all dependencies bundled then Docker would make sense.
There artifact is a single statically linked binary/executable, not a docker container. They can build that binary once and pass it along a deployment pipeline i.e dev, test, prod changing config parameters for each environment but the exact same code running.
Systemd just like the various container runtimes supports auto restarts + logging. You can have the same alerting tools hang off your logs etc etc.
The fact they are not using Docker does not mean they can't have a proper build/deployment pipeline. The fact they are dealing with a single static executable makes building a pipeline and robust deployment far simpler as they have far fewer moving pieces.
If the author was deploying python, javascript, ruby apps where you don't get a static executable artifact or fat jar with all dependencies bundled then Docker would make sense.
I've been struggling with this for years now. Every time I ask the question "why do I need Docker when Go produces static binary executables?" I get some reasonable answers, but nothing along the lines of "you can't do X without Docker".
I totally grok the need when your deployable is a few hundred script files for a very specific runtime and a large set of very exact dependencies. But that's not the situation with Go.
And now //embed too. Adding all my templates, static files, everything, to the binary. Ship one file. Awesome
I totally grok the need when your deployable is a few hundred script files for a very specific runtime and a large set of very exact dependencies. But that's not the situation with Go.
And now //embed too. Adding all my templates, static files, everything, to the binary. Ship one file. Awesome
Yeah, there isn't really anything that you just can't do at all without Docker. The question is whether it's a net positive or negative on your architecture as a whole. Eg, I deploy some Go apps using docker (basically build the executable and then build a docker image that just contains that executable). Looking at just the single application, it's pure overhead to add Docker vs just deploying the binary executable somewhere and running it. But in the overall context of my setup, since I'm running other apps as well that are written in other languages/frameworks and have different characteristics, it's a huge net positive for me to have a single uniform interface for deploying and running them. Everything gets pushed to the same container registry the same way, is versioned the same way, can handle service discovery the same way (by listening on the docker port), can do canary deploys the same way, etc. I can use docker-compose to bring up a dev environment with multiple services with a single command. I can deploy the container to Cloud Run or ECS or an equivalent if that makes more sense than running servers.
I've just being doing this process for my product now. I have a bunch of deployment scripts that control the test, build and deploy of the app.
I can't run them on a Docker container (the final hurdle was that Docker can't run systemd). So my choice was to either add Docker to my production servers, or drop Docker and use Vagrant instead for localhost VM for dev.
Again, I couldn't see what Docker was adding to the mix that was of value - it would be an additional layer of configuration, complexity and failure on the production servers. It wouldn't save anything if the server went down, it would complicate attempts to restart the app if that crashed, and it gives us... what?
Again, I get it for Rails or Django builds (and similar) where the environment is complex and dependencies have to be managed carefully and can conflict horribly. But I just don't have that problem. And it's a real incentive to stick to the one language and not introduce any more dependencies ;)
I can't run them on a Docker container (the final hurdle was that Docker can't run systemd). So my choice was to either add Docker to my production servers, or drop Docker and use Vagrant instead for localhost VM for dev.
Again, I couldn't see what Docker was adding to the mix that was of value - it would be an additional layer of configuration, complexity and failure on the production servers. It wouldn't save anything if the server went down, it would complicate attempts to restart the app if that crashed, and it gives us... what?
Again, I get it for Rails or Django builds (and similar) where the environment is complex and dependencies have to be managed carefully and can conflict horribly. But I just don't have that problem. And it's a real incentive to stick to the one language and not introduce any more dependencies ;)
In my opinion whether those are red flags depends entirely on the context. How many people work on the project, code base size and age, etc.
I feel these days projects are often starting out with way too much complexity and shiny tools. It should not be about cutting out things, but instead about adding things at the point they really add value.
I feel these days projects are often starting out with way too much complexity and shiny tools. It should not be about cutting out things, but instead about adding things at the point they really add value.
No of the points you list have anything to do with docker.
1. No I don't, I still would potentially not use Docker in many cases (but I would use a CI, which might or might not run in a docker image, but it's not the same as deploying docker images).
2. Depends on the language I'm using, for some languages I would be afraid of accidental incompatibilities. For others I'm not worried and would be fine if roughly the same OS is used in CI and production.
3. Can happen with docker too, on the other hand VM or auto non-VM restarts exists independent of Docker. I'm not sure why you mention systemd here, it has very reasonable "auto restart if not alive" features.
Through then in the end I'm increasingly drifting more to use images, but I really don't want to use docker in production. But then I can do what people normally expect from "using docker" without docker, e.g. by using podman or other less "root" heavy ways to run VM's with appropriate tooling for reliability (which yes can be systemd+rootless podman in some cases).
1. No I don't, I still would potentially not use Docker in many cases (but I would use a CI, which might or might not run in a docker image, but it's not the same as deploying docker images).
2. Depends on the language I'm using, for some languages I would be afraid of accidental incompatibilities. For others I'm not worried and would be fine if roughly the same OS is used in CI and production.
3. Can happen with docker too, on the other hand VM or auto non-VM restarts exists independent of Docker. I'm not sure why you mention systemd here, it has very reasonable "auto restart if not alive" features.
Through then in the end I'm increasingly drifting more to use images, but I really don't want to use docker in production. But then I can do what people normally expect from "using docker" without docker, e.g. by using podman or other less "root" heavy ways to run VM's with appropriate tooling for reliability (which yes can be systemd+rootless podman in some cases).
I don’t think any of these need docker though?
You can copy binaries to/from staging just fine.
They have a CI job that builds and pushes code. So it does not matter what the new guy did on his laptop.
I am not sure if you are going for “monitoring” or “redundancy” in your dead VM example, but docker by itself cannot provide either of those. You need some solution either way.
I use docker daily because our app is big and has tons of system dependencies, but I have no other choice. We need fast version rollback/update and regular approach of installing deb packages will not work well. I dream of nixOS and sylabs, but my org is not going to switch to radical new technology. But if someone can set up the system so they don’t need docker - more power to them, I can only be envious.
You can copy binaries to/from staging just fine.
They have a CI job that builds and pushes code. So it does not matter what the new guy did on his laptop.
I am not sure if you are going for “monitoring” or “redundancy” in your dead VM example, but docker by itself cannot provide either of those. You need some solution either way.
I use docker daily because our app is big and has tons of system dependencies, but I have no other choice. We need fast version rollback/update and regular approach of installing deb packages will not work well. I dream of nixOS and sylabs, but my org is not going to switch to radical new technology. But if someone can set up the system so they don’t need docker - more power to them, I can only be envious.
> I dream of nixOS and sylabs, but my org is not going to switch to radical new technology.
From https://en.wikipedia.org/wiki/NixOS
> Initial release 2003; 18 years ago
From https://en.wikipedia.org/wiki/Docker_(software)
> Initial release March 20, 2013; 7 years ago
;)
From https://en.wikipedia.org/wiki/NixOS
> Initial release 2003; 18 years ago
From https://en.wikipedia.org/wiki/Docker_(software)
> Initial release March 20, 2013; 7 years ago
;)
It is not NixOS vs Docker, it is NixOS vs Docker/Ubuntu.
Most of the day-to-day OS problems don't come from Docker, they come from base linux distribution. And Ubuntu was released in Oct 2004, and it is in big part Debian, which was released in 1993.
There is a big advantage when you run the same OS on developers' desktops and on server. It is also great when all sorts of weird third-party packages already know and support your OS.
This is the advantage of Docker I suppose -- it does not "get in your way" and lets you use the same things you used to do before it.
Most of the day-to-day OS problems don't come from Docker, they come from base linux distribution. And Ubuntu was released in Oct 2004, and it is in big part Debian, which was released in 1993.
There is a big advantage when you run the same OS on developers' desktops and on server. It is also great when all sorts of weird third-party packages already know and support your OS.
This is the advantage of Docker I suppose -- it does not "get in your way" and lets you use the same things you used to do before it.
Note that I only linked to the NixOS page since the Nix page doesn't have an "Initial release" entry ( https://en.wikipedia.org/wiki/Nix_package_manager ).
I've personally used Nix on Ubuntu, NixOS and macOS. Anecdotally, I can't get Docker to work on my Mac :(
I've personally used Nix on Ubuntu, NixOS and macOS. Anecdotally, I can't get Docker to work on my Mac :(
> I am not sure if you are going for “monitoring” or “redundancy” in your dead VM example, but docker by itself cannot provide either of those. You need some solution either way.
You are the second one that say this here... that explain why Docker Swarm lose traction versus Kubernetes, that's a makerting issue.
I just setup a Docker Swarm cluster recently, in 5 minutes it was redundant. I didn't add any other software, just the basic docker swarm command that shipped with it.
I actually didn't needed the redundancy part at all, but I wanted a second server that could ping the first one and send me an alert. I was going to simply put it on a machine, with systemd like him, but it was just as easy to run it on the machine using Docker. Hell I could run it on both even more easily than doing that twice using systemd... I don't even know how to use systemd now that I mention it....
You are the second one that say this here... that explain why Docker Swarm lose traction versus Kubernetes, that's a makerting issue.
I just setup a Docker Swarm cluster recently, in 5 minutes it was redundant. I didn't add any other software, just the basic docker swarm command that shipped with it.
I actually didn't needed the redundancy part at all, but I wanted a second server that could ping the first one and send me an alert. I was going to simply put it on a machine, with systemd like him, but it was just as easy to run it on the machine using Docker. Hell I could run it on both even more easily than doing that twice using systemd... I don't even know how to use systemd now that I mention it....
You're not worried that Swarm will get abandoned by Docker? (Or already is?)
[deleted]
Millions of transactions per second is not the point of using docker.
Docker is about running multiple things on the same server. If you have 1 server and you have to run 1 app on it, you don't need a Docker container.
Now if you have physical server and you want to use it fully you want to run ALL kinds of apps on it. What is hard is that running multiple apps, you run into issues with dependencies. Ideally you run each app on different server or separate OS then you don't have that problem, but that is costly and if one app sits idle you waste money you spent on the server.
I don't know why people think about it the other way that you get 1 app and run it distributed is "peak containers". Peak containers is about having common infra so you make the most of your server because you don't care what app is running in there, your server does not need to have any special config for your special app. You don't get bogged down that you cannot install anything else on that server because of dependency hell.
Well from the application point of view bonus is that if you dockerize your app you don't care about the servers it is running on. You just ship it and it will run on someones server with other apps that you don't know about and don't care about and their dependencies will not break your app.
Docker is about running multiple things on the same server. If you have 1 server and you have to run 1 app on it, you don't need a Docker container.
Now if you have physical server and you want to use it fully you want to run ALL kinds of apps on it. What is hard is that running multiple apps, you run into issues with dependencies. Ideally you run each app on different server or separate OS then you don't have that problem, but that is costly and if one app sits idle you waste money you spent on the server.
I don't know why people think about it the other way that you get 1 app and run it distributed is "peak containers". Peak containers is about having common infra so you make the most of your server because you don't care what app is running in there, your server does not need to have any special config for your special app. You don't get bogged down that you cannot install anything else on that server because of dependency hell.
Well from the application point of view bonus is that if you dockerize your app you don't care about the servers it is running on. You just ship it and it will run on someones server with other apps that you don't know about and don't care about and their dependencies will not break your app.
> Docker is about running multiple things on the same server
Or about running a single thing on multiple servers. Or providing a temporary runtime with language/lib/software X preinstalled. Or to facilitate easier testing.
In the end, many people use containers for many different reasons. There isn't a single "This is what docker is for", although people tend to shove containers into places that might not be 100% suitable for containers.
Or about running a single thing on multiple servers. Or providing a temporary runtime with language/lib/software X preinstalled. Or to facilitate easier testing.
In the end, many people use containers for many different reasons. There isn't a single "This is what docker is for", although people tend to shove containers into places that might not be 100% suitable for containers.
> There isn't a single "This is what docker is for"
100% this. Choosing the right tool for the job, that's what distinguishes developers from engineers in my opinion.
There is place for almost every tool be it leftpad npm package or running go on bare metal.
Most sensationalist articles, "Why we don't use [popular tool]" just seem to have usecase that doesn't fit this particular tool.
100% this. Choosing the right tool for the job, that's what distinguishes developers from engineers in my opinion.
There is place for almost every tool be it leftpad npm package or running go on bare metal.
Most sensationalist articles, "Why we don't use [popular tool]" just seem to have usecase that doesn't fit this particular tool.
> What is hard is that running multiple apps, you run into issues with dependencies.
That's exactly the point they were making in this article. Also even with 1 server and 1 app people still use Docker because the pain of setting up a Linux machine with all of the dependencies for your average Linux software is so great, and Docker simplifies that.
The single static binary is such a great feature of Go. I wish more languages would copy it.
That's exactly the point they were making in this article. Also even with 1 server and 1 app people still use Docker because the pain of setting up a Linux machine with all of the dependencies for your average Linux software is so great, and Docker simplifies that.
The single static binary is such a great feature of Go. I wish more languages would copy it.
> Now if you have physical server and you want to use it fully you want to run ALL kinds of apps on it.
There are so many options other than Docker as of today. These provide very different tradeoffs. Somehow we got to the point that people think that there are only one solution (Docker) to run multiple applications on a single server.
Some other options:
- single binary packaging
- LXC / LXD
- Firecracker
- chroot
There are so many options other than Docker as of today. These provide very different tradeoffs. Somehow we got to the point that people think that there are only one solution (Docker) to run multiple applications on a single server.
Some other options:
- single binary packaging
- LXC / LXD
- Firecracker
- chroot
Why not just run VMs? If you have a few VM images you can deploy - isn't it the same benefit for less work?
Docker has tooling for building images and is lighter than VM.
Devs can make their docker containers quite easily. With VM you have to update operating system and other stuff. With Docker as a Developer you are concerned only about your direct dependencies.
I run my systems in VMs and it is quite a lot of work to update all unrelated stuff. Well related stuff is also a lot of work but that is price I am willing to pay for.
Devs can make their docker containers quite easily. With VM you have to update operating system and other stuff. With Docker as a Developer you are concerned only about your direct dependencies.
I run my systems in VMs and it is quite a lot of work to update all unrelated stuff. Well related stuff is also a lot of work but that is price I am willing to pay for.
I use Nix for building projects and managing my laptops (NixOS at home, nix-darwin on macOS at work). This solves the 'dependency hell' in a much cleaner, more efficient and IMHO simpler way than docker.
For those who don't know, Nix doesn't install anything system-wide. Everything gets prefixed with a hash of its dependencies (e.g. /nix/store/cfh2830k2p1gg10j40x27rmlmpwqy7p4-git-2.23.1/bin/git). Since nothing is available system-wide, program A can only use program B by specifying its whole path (including the hash); this affects the hash of program A, and so on for anything that refers to it. This is a Merkle tree, giving the provenance of every dependency, all the way down to the 'bootstrap' tools (e.g. a prebuilt GCC that was used to build the GCC that was used to build the tar that was used to extract the bash source ....). Here's a snippet of the dependencies used to build the above git command:
The Guix project takes this a bit further, with its `guix pack` command turning paths like this into a runnable docker image (amongst other options).
Compare this to the way docker is used: we download and run some large binary blob ('image'), containing who-knows-what. These images are usually bloated with all sorts os unneeded cruft (due to the way docker builds images 'from the inside'); often a whole OS, package management tools, or at the very least a shell like busybox, all of which makes scaling harder and increases our attack surface. The build steps for those images are often insecure and unreproducible too, e.g. running commands like `apt-get install -y foo` or `pip install bar`, whose behaviour changes over time dependending on the contents of third-party servers.
I really want to start using containers for their security and deployment advantages, but so much of their tooling and approach seems completely backwards to me :(
(Note that it's possible to build docker images using Nix, but this requires running commands like 'docker load' which I can't get working on my Mac, and deploying to AWS ECS seems to require 'image registries' rather than just pointing to a file on S3. Urgh.)
For those who don't know, Nix doesn't install anything system-wide. Everything gets prefixed with a hash of its dependencies (e.g. /nix/store/cfh2830k2p1gg10j40x27rmlmpwqy7p4-git-2.23.1/bin/git). Since nothing is available system-wide, program A can only use program B by specifying its whole path (including the hash); this affects the hash of program A, and so on for anything that refers to it. This is a Merkle tree, giving the provenance of every dependency, all the way down to the 'bootstrap' tools (e.g. a prebuilt GCC that was used to build the GCC that was used to build the tar that was used to extract the bash source ....). Here's a snippet of the dependencies used to build the above git command:
$ nix-store -q --requisites $(nix-store --deriver /nix/store/cfh2830k2p1gg10j40x27rmlmpwqy7p4-git-2.23.1)
/nix/store/01n3wxxw29wj2pkjqimmmjzv7pihzmd7-which-2.21.tar.gz.drv
/nix/store/064jmylcq7h6fa5asg0rna9awcqz3765-0001-x86-Properly-merge-GNU_PROPERTY_X86_ISA_1_USED.patch
/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh
/nix/store/na8n4ndbmkc500jm7argsprzj94p27h4-libev-4.27.tar.gz.drv
/nix/store/33sl3bqjcqzrdd9clgaad3ljlwyl1pkb-patch-shebangs.sh
/nix/store/81ikflgpwzgjk8b5vmvg9gaw9mbkc86k-compress-man-pages.sh
/nix/store/9ny6szla9dg61jv8q22qbnqsz37465n0-multiple-outputs.sh
/nix/store/a92kz10cwkpa91k5239inl3fd61zp5dh-move-lib64.sh
/nix/store/5g9j02amqihyg9b15m9bds9iqkjclcgr-bootstrap-tools.tar.xz.drv
/nix/store/b7irlwi2wjlx5aj1dghx4c8k3ax6m56q-busybox.drv
/nix/store/c0sr4qdy8halrdrh5dpm7hj05c6hyssa-unpack-bootstrap-tools.sh
/nix/store/dccfayc70jsvzwq023smvqv04c57bsib-bootstrap-tools.drv
...
$ nix-store -q --requisites /nix/store/cfh2830k2p1gg10j40x27rmlmpwqy7p4-git-2.23.1
/nix/store/5gga7b7aavb4vcnia44mzd6m9vrx46gq-perl5.30.0-LWP-MediaTypes-6.04
/nix/store/9n0n6dg310azjmnyfxq6bsn0jsz9cvk8-perl5.30.0-URI-1.76
/nix/store/9zmmc8akdb2jlkapqf9p26jn84xvzbjr-perl5.30.0-HTTP-Date-6.02
/nix/store/ds92hvb2jx8yfh2j3wqy1fqr07654nkr-perl5.30.0-Encode-Locale-1.05
/nix/store/l43m8p3bkqar54krnj3jalbim7m64lid-perl5.30.0-IO-HTML-1.001
/nix/store/zbd643lb2a8iqgp1m36r1adzpqpwhr7s-perl5.30.0-HTTP-Message-6.18
/nix/store/0hkq3crf7357z5zywx4i9dkddi0qvb8k-perl5.30.0-HTTP-Daemon-6.01
/nix/store/5ka41zhii1bjss3f60rzd2npz9mxj060-glibc-2.27
/nix/store/7fvwr8la2k701hrx2w5xnvjr5kkc7ysv-gcc-8.3.0-lib
/nix/store/clp0hgzq3pwx03lr8apg92nmswr4izvz-bash-4.4-p23
/nix/store/hgdh73wvrncfnmdmg1damcfrviv9cgp7-gnum4-1.4.18
/nix/store/ig8wzhnxxxrspcyb85wxqdbbn2q53088-bison-3.4.2
/nix/store/11z0140njah4dbrngg5bcx99nyw391kw-gettext-0.19.8.1
/nix/store/1ycvwhslagg7dn8rpc3simyfkirgjbp5-perl5.30.0-Net-HTTP-6.19
/nix/store/n3ja6nl5i91x5sn232f5fyhmx7lxcmj1-ncurses-6.1-20190112
/nix/store/47w0y1vavaxja3mm9gr5dmbgxrjgpw21-libedit-20190324-3.1
/nix/store/84jxhr8l3plkd6z2x4v941za9kvmv88g-zlib-1.2.11
/nix/store/dfgm23z5mvp7i3085dlq4rn0as41jp1p-openssl-1.1.1d
...
The Guix project takes this a bit further, with its `guix pack` command turning paths like this into a runnable docker image (amongst other options).
Compare this to the way docker is used: we download and run some large binary blob ('image'), containing who-knows-what. These images are usually bloated with all sorts os unneeded cruft (due to the way docker builds images 'from the inside'); often a whole OS, package management tools, or at the very least a shell like busybox, all of which makes scaling harder and increases our attack surface. The build steps for those images are often insecure and unreproducible too, e.g. running commands like `apt-get install -y foo` or `pip install bar`, whose behaviour changes over time dependending on the contents of third-party servers.
I really want to start using containers for their security and deployment advantages, but so much of their tooling and approach seems completely backwards to me :(
(Note that it's possible to build docker images using Nix, but this requires running commands like 'docker load' which I can't get working on my Mac, and deploying to AWS ECS seems to require 'image registries' rather than just pointing to a file on S3. Urgh.)
Totally unrelated question: Do Nix packages typically come with proper license & copyright notes (and source code, if necessary) and is there an easy way to extract them for the entire dependency tree?
The reason I'm asking is that, as our team is getting closer to ship our $PRODUCT, we started worrying about the licenses of all the third-party software components we use. Needless to say, we're using some Docker image as base and now need to figure out what software it contains exactly…
The reason I'm asking is that, as our team is getting closer to ship our $PRODUCT, we started worrying about the licenses of all the third-party software components we use. Needless to say, we're using some Docker image as base and now need to figure out what software it contains exactly…
Yes, packages in Nixpkgs (the official repository of Nix packages) have mandatory copyright or license information.
This can also be accessed programmatically, if you intend to scan all your dependencies:
$ nix-env -qaA nixpkgs.vim --json | jq '."nixpkgs.vim".meta.license'
{
"fullName": "Vim License",
"shortName": "vim",
"spdxId": "Vim",
"url": "https://spdx.org/licenses/Vim.html"
}
In addition to what rnhmjoj says, Nix can also check the license of packages as it's evaluating them (i.e. before fetching/building). The most obvious place this appears is the `allowUnfree` option, which defaults to `false` and refuses to evaluate proprietary packages. Details on how to whitelist/blacklist specific licenses, and how to define custom functions to decide this, are given at https://nixos.org/manual/nixpkgs/stable/#sec-allow-unfree
> and source code, if necessary
Nix is completely based around source code, similar to Gentoo's emerge, BSD ports, etc. (in contrast to those based around binaries, like dpkg, RPM, etc.).
More precisely, Nix works with "derivations", which have outputs (the paths in /nix/store which they define), inputs (the outputs of other derivations), and a "builder" (this is usually bash, with a build script given as argument).
The files above which end in `.drv` define derivations. Fun fact: Nix doesn't care how these files are made; we can write our own tools to create .drv files, which is exactly what Guix and hnix do :)
Notice that the runtime dependencies above don't contain any .drv files. That's because at runtime we only need (some of) the outputs of (some of) the derivations.
Also note that the build dependencies contain lots of .drv files, since the build instructions are just as much a part of the 'source' as anything else.
Some of those derivations have names like `.tar.gz.drv` and `.xz.drv`; those define how to fetch a particular archive, e.g. containing a URL, maybe a bunch of mirrors, a SHA256 hash to compare it against, etc. Internally, Nix doesn't distinguish between such "sources" and anything else; it's all just derivations.
Since Nix builds tend to be reproducible (although that's not guaranteed), it's able to use a clever optimisation trick: packages (i.e. the outputs of derivations) are completely specified by their hash, so we can copy them from someone else (if they already have it), rather than building them ourselves, and the result should be the same. Similar to how we can checkout a particular git commit from someone who already has it, rather than having to recreate the tree ourselves by make all the same edits in the same order. (It's not exactly the same, since we need to trust that person isn't sending us something malicious).
Hence Nix can use a (trusted) binary cache: this is just a folder full of derivation outputs, which we can query to see if our desired output has already been built, before we bother doing it ourselves.
Binary caches are the reason I can do `nix-shell -p git` and have a git binary ready to go in a couple of seconds, despite that binary being defined in terms of those massive source-based dependency graphs.
Nix is completely based around source code, similar to Gentoo's emerge, BSD ports, etc. (in contrast to those based around binaries, like dpkg, RPM, etc.).
More precisely, Nix works with "derivations", which have outputs (the paths in /nix/store which they define), inputs (the outputs of other derivations), and a "builder" (this is usually bash, with a build script given as argument).
The files above which end in `.drv` define derivations. Fun fact: Nix doesn't care how these files are made; we can write our own tools to create .drv files, which is exactly what Guix and hnix do :)
Notice that the runtime dependencies above don't contain any .drv files. That's because at runtime we only need (some of) the outputs of (some of) the derivations.
Also note that the build dependencies contain lots of .drv files, since the build instructions are just as much a part of the 'source' as anything else.
Some of those derivations have names like `.tar.gz.drv` and `.xz.drv`; those define how to fetch a particular archive, e.g. containing a URL, maybe a bunch of mirrors, a SHA256 hash to compare it against, etc. Internally, Nix doesn't distinguish between such "sources" and anything else; it's all just derivations.
Since Nix builds tend to be reproducible (although that's not guaranteed), it's able to use a clever optimisation trick: packages (i.e. the outputs of derivations) are completely specified by their hash, so we can copy them from someone else (if they already have it), rather than building them ourselves, and the result should be the same. Similar to how we can checkout a particular git commit from someone who already has it, rather than having to recreate the tree ourselves by make all the same edits in the same order. (It's not exactly the same, since we need to trust that person isn't sending us something malicious).
Hence Nix can use a (trusted) binary cache: this is just a folder full of derivation outputs, which we can query to see if our desired output has already been built, before we bother doing it ourselves.
Binary caches are the reason I can do `nix-shell -p git` and have a git binary ready to go in a couple of seconds, despite that binary being defined in terms of those massive source-based dependency graphs.
"sandbox" is Docker's bread and butter
Small docker image: look into Alpine which comes in at ~5MB. You are basically starting from nothing.
Versioning: both apt and pip allow you to specify a version. I dont think this is a docker-specific issue. I'm sure there are docker images with old packages on them. Or i suppose you could just run a tool that downloads specific versions of binaries (like nix) in your Dockerfile.
Small docker image: look into Alpine which comes in at ~5MB. You are basically starting from nothing.
Versioning: both apt and pip allow you to specify a version. I dont think this is a docker-specific issue. I'm sure there are docker images with old packages on them. Or i suppose you could just run a tool that downloads specific versions of binaries (like nix) in your Dockerfile.
> "sandbox" is Docker's bread and butter
Absolutely. I would ultimately like a container image (docker format or otherwise), containing precisely the files that are needed by the intended service, and nothing else. In particular, the container should not contain unwanted cruft like wget, pip, apt, vi, cp, etc.; and it certainly shouldn't contain a shell (bash/dash/busybox/etc.).
The way docker runs containers is fine (albeit overly-complicated for my tastes, e.g. "loading an image from a registry", compared to something like `kvm -hda myImage.img`)
The way docker's tools build those images is bad. Hence why Nix, Guix, etc. are better for that.
> Small docker image: look into Alpine which comes in at ~5MB. You are basically starting from nothing.
Nope, that is starting from an entire OS. Even a busybox-only image is made dangerous by the existence of a shell, since we have to worry about exploits letting attackers 'shell out' (and from there, inspecting and altering the application binary, not to mention trying to jailbreak or remote-out of the shell to do damage elsewhere, etc.)
> Versioning: both apt and pip allow you to specify a version.
I don't particularly care about versions; I want to specify the SHA hash, so I know if something has changed on the third party server (pypi, ubuntu, or whatever); or whether my connection has been re-routed maliciously; or the download was interrupted/corrupted; or some distant config/setting has changed the resolver's behaviour; and so on.
I also want some way to enforce this, so that no file can exist in the container without being verified against an explicit SHA: either directly, or extracted from an archive with a specified SHA, or checked-out of a particular git commit (which is itself a SHA), or built in a deterministic/reproducible way from components with known SHAs (e.g. a tarball, GCC binary and Makefile, all with known SHAs), etc. I want a fatal error if I forget a SHA.
A system like that would be tolerable, but still an uncomfortable step backwards.
Verifying SHAs is good, but we'd still be running random binaries fetched from a third-party. Unlike apt, pip, etc. Nix gives me the full provenance of everything: I can, at a glance, see whether a particular patch was applied to the GCC that compiled the bash that was used to build my binary; or whatever (admittedly, this is more useful for auditing things like OpenSSL, rather than trusting-trust attacks on GCC).
I can also, with a single command, recreate any of those build environments in a shell, e.g. if I need to test the presence of a vulnerability or bug. As a by-product, I can also use those for development, jumping in to any step along the provenance chain to edit files, run tests, etc.
I can also choose whether or not I want to trust the builder/provider of those binaries, or whether I want to build them myself, and I can choose at what level I want to place that trust; e.g. maybe I trust their GCC, but I want to build the JDK myself. As a by-product, I can also switch out or alter any of those steps, e.g. enabling a GCC flag for a certain build, completely replacing the JDK with another, etc. The Nix language, and the architecture of the Nixpkgs repository, makes this very easy; and Nix itself takes care of rebuilding everything downstream of such changes.
To understand this, I like to think of Nix more like an alternative to Make, rather than apt/dpkg/pip/etc. Those tools are band-aids, created because Make doesn't scale or compose very well. Similarly, Dockerfiles/Chef/Puppet/Ansible/etc. are band-aids created because apt/dpkg/pip/etc. don't scale or compose well. Nix does compose and scale well, so it can be used across that whole spectrum of 'get result FOO by running commands BAR' (we can take it even further with projects like Disnix and NixOps which perform orchestration, but I've personally never used them).
Absolutely. I would ultimately like a container image (docker format or otherwise), containing precisely the files that are needed by the intended service, and nothing else. In particular, the container should not contain unwanted cruft like wget, pip, apt, vi, cp, etc.; and it certainly shouldn't contain a shell (bash/dash/busybox/etc.).
The way docker runs containers is fine (albeit overly-complicated for my tastes, e.g. "loading an image from a registry", compared to something like `kvm -hda myImage.img`)
The way docker's tools build those images is bad. Hence why Nix, Guix, etc. are better for that.
> Small docker image: look into Alpine which comes in at ~5MB. You are basically starting from nothing.
Nope, that is starting from an entire OS. Even a busybox-only image is made dangerous by the existence of a shell, since we have to worry about exploits letting attackers 'shell out' (and from there, inspecting and altering the application binary, not to mention trying to jailbreak or remote-out of the shell to do damage elsewhere, etc.)
> Versioning: both apt and pip allow you to specify a version.
I don't particularly care about versions; I want to specify the SHA hash, so I know if something has changed on the third party server (pypi, ubuntu, or whatever); or whether my connection has been re-routed maliciously; or the download was interrupted/corrupted; or some distant config/setting has changed the resolver's behaviour; and so on.
I also want some way to enforce this, so that no file can exist in the container without being verified against an explicit SHA: either directly, or extracted from an archive with a specified SHA, or checked-out of a particular git commit (which is itself a SHA), or built in a deterministic/reproducible way from components with known SHAs (e.g. a tarball, GCC binary and Makefile, all with known SHAs), etc. I want a fatal error if I forget a SHA.
A system like that would be tolerable, but still an uncomfortable step backwards.
Verifying SHAs is good, but we'd still be running random binaries fetched from a third-party. Unlike apt, pip, etc. Nix gives me the full provenance of everything: I can, at a glance, see whether a particular patch was applied to the GCC that compiled the bash that was used to build my binary; or whatever (admittedly, this is more useful for auditing things like OpenSSL, rather than trusting-trust attacks on GCC).
I can also, with a single command, recreate any of those build environments in a shell, e.g. if I need to test the presence of a vulnerability or bug. As a by-product, I can also use those for development, jumping in to any step along the provenance chain to edit files, run tests, etc.
I can also choose whether or not I want to trust the builder/provider of those binaries, or whether I want to build them myself, and I can choose at what level I want to place that trust; e.g. maybe I trust their GCC, but I want to build the JDK myself. As a by-product, I can also switch out or alter any of those steps, e.g. enabling a GCC flag for a certain build, completely replacing the JDK with another, etc. The Nix language, and the architecture of the Nixpkgs repository, makes this very easy; and Nix itself takes care of rebuilding everything downstream of such changes.
To understand this, I like to think of Nix more like an alternative to Make, rather than apt/dpkg/pip/etc. Those tools are band-aids, created because Make doesn't scale or compose very well. Similarly, Dockerfiles/Chef/Puppet/Ansible/etc. are band-aids created because apt/dpkg/pip/etc. don't scale or compose well. Nix does compose and scale well, so it can be used across that whole spectrum of 'get result FOO by running commands BAR' (we can take it even further with projects like Disnix and NixOps which perform orchestration, but I've personally never used them).
> Now if you have physical server and you want to use it fully you want to run ALL kinds of apps on it. What is hard is that running multiple apps, you run into issues with dependencies. Ideally you run each app on different server or separate OS then you don't have that problem, but that is costly and if one app sits idle you waste money you spent on the server.
I mean yeah, it was sortof a problem on late 90s x86 kit. But its not been a real problem for a good 20 years. virtualisation solved that issue. Modern processors (well not even modern ones, any xeon from the last 10 years) will have virtualisation optimization in them.
Docker allows you to not be as disciplined when it comes to your build environment. This means that you can let developers ship any old shit. This will give you a short term productivity boost, and longterm tech debt.
> your server does not need to have any special config for your special app
yeahna, thats not really true. You still need to manage secrets (docker has no real security bits built in, barring cgroups) you still have to manage volumes, and you still have to manage connectivity. Not to mention resource discovery.
The biggest lie that K8s and docker told us is that you didn't need a coherent plan for your infra. To be efficient, you need to make your apps work on the same dependencies, same platform, and re-use as much as possible.
I mean yeah, it was sortof a problem on late 90s x86 kit. But its not been a real problem for a good 20 years. virtualisation solved that issue. Modern processors (well not even modern ones, any xeon from the last 10 years) will have virtualisation optimization in them.
Docker allows you to not be as disciplined when it comes to your build environment. This means that you can let developers ship any old shit. This will give you a short term productivity boost, and longterm tech debt.
> your server does not need to have any special config for your special app
yeahna, thats not really true. You still need to manage secrets (docker has no real security bits built in, barring cgroups) you still have to manage volumes, and you still have to manage connectivity. Not to mention resource discovery.
The biggest lie that K8s and docker told us is that you didn't need a coherent plan for your infra. To be efficient, you need to make your apps work on the same dependencies, same platform, and re-use as much as possible.
I use docker with go all the time.
The only thing in container images I build are static binaries regardless of if it is C, rust, go, whatever. Static binaries don't defeat the point of containers, they are best practice. People that include bash and a package manager and such nonsense inside production containers huge complex and insecure for no reason.
The problem containers solve better than anything else are providing a simple API to take advantage of kernel security features that are a real pain to use otherwise.
User, process, and filesystem namespacing. System call filtering. Ensuring even if there is a remote exec exploit there is no bash or anything else in the container for an attacker to use.
If you think containers are only a software build and distribution tool then security is probably not even part of your thought process, and that is dangerous.
Ignore all the 99% of blog posts and tuturials on containers out there written by people who have never done any kind of system hardening.
They are mostly making your life harder and giving you nothing in return, as the author of this post points to.
The only thing in container images I build are static binaries regardless of if it is C, rust, go, whatever. Static binaries don't defeat the point of containers, they are best practice. People that include bash and a package manager and such nonsense inside production containers huge complex and insecure for no reason.
The problem containers solve better than anything else are providing a simple API to take advantage of kernel security features that are a real pain to use otherwise.
User, process, and filesystem namespacing. System call filtering. Ensuring even if there is a remote exec exploit there is no bash or anything else in the container for an attacker to use.
If you think containers are only a software build and distribution tool then security is probably not even part of your thought process, and that is dangerous.
Ignore all the 99% of blog posts and tuturials on containers out there written by people who have never done any kind of system hardening.
They are mostly making your life harder and giving you nothing in return, as the author of this post points to.
> User, process, and filesystem namespacing. System call filtering. Ensuring even if there is a remote exec exploit there is no bash or anything else in the container for an attacker to use.
All of these are supported by systemd out of the box. You can use `SystemCallFilter=...`, and various options to restrict the file system (like `ProtectHome` and `ProtectSystem`) plus tons of other linux kernel features (capabilities, no new privileges).
All of these are supported by systemd out of the box. You can use `SystemCallFilter=...`, and various options to restrict the file system (like `ProtectHome` and `ProtectSystem`) plus tons of other linux kernel features (capabilities, no new privileges).
Fair point, but there is also no portable spec for this that works beyond systemd and there are plenty of cases where systemd is not desirable.
Maybe you want to run on an embedded system, or a hardened immutable system where you don't actually need bash and coreutils etc.
The OCI spec gives us all these features in a compact and portable package independent of any particular init system while also providing a way to verify and update images with signature verification over a network so you need not even have attack surface like ssh to manage updates.
Don't get me wrong, I love systemd for my desktop, however for servers I don't want a single binary present that isn't strictly nessesary for my target binary to function as it just gives adversaries a foothold.
Maybe you want to run on an embedded system, or a hardened immutable system where you don't actually need bash and coreutils etc.
The OCI spec gives us all these features in a compact and portable package independent of any particular init system while also providing a way to verify and update images with signature verification over a network so you need not even have attack surface like ssh to manage updates.
Don't get me wrong, I love systemd for my desktop, however for servers I don't want a single binary present that isn't strictly nessesary for my target binary to function as it just gives adversaries a foothold.
There’s a certain section of HN that has apparently decided that everything was better when C was considered a high level language, production was a single physical machine that you personally managed the cabling for, and 640Kb of memory was all anybody would ever need. This section of HN is either composed of god-tier ninja programmers, or of people looking back through rose-tinted specs and pining for “the good old days” - my guess is mostly the latter.
Maybe you run an amazing shop and everyone should be as good as you - I’ll give you the benefit of the doubt. But in my experience, every company I’ve seen that’s populated by people with this kind of attitude has been a raging dumpster fire, perpetually running a hair’s breadth from exploding violently.
The world writes, reads, and runs a lot more software than it used to, serving a lot more people in a lot more countries with a lot more variation in the circumstances of how it’s used. This adds up to a lot more complexity.
Docker is one very important tool that helps us to manage this complexity. Yes, it also introduces some difficulties that you didn’t have before, but it solves much more difficult problems pretty well. We use docker for almost everything, and it’s a lifesaver.
Maybe you run an amazing shop and everyone should be as good as you - I’ll give you the benefit of the doubt. But in my experience, every company I’ve seen that’s populated by people with this kind of attitude has been a raging dumpster fire, perpetually running a hair’s breadth from exploding violently.
The world writes, reads, and runs a lot more software than it used to, serving a lot more people in a lot more countries with a lot more variation in the circumstances of how it’s used. This adds up to a lot more complexity.
Docker is one very important tool that helps us to manage this complexity. Yes, it also introduces some difficulties that you didn’t have before, but it solves much more difficult problems pretty well. We use docker for almost everything, and it’s a lifesaver.
These people aren’t arguing that docker is a bad tool to manage complexity, they’re arguing that a lot of the time we build things that are unnecessarily complex. Most “scaling” issues are just performance bottlenecks that can be addressed with a profiler and some optimisation, yet the go-to answer nowadays seems to be to just throw more machines at the problem.
Probably because more machines is faster and cheaper than a profiler.
At least in the short term.
At least in the short term.
If a programmer is not able to write efficient code, they may not even know that their code is inefficient.
I regularly see programmers that come from a front end or "full stack" background try to solve data engineering problems, and end up with solutions that are 10x, 100x or sometimes 1000x slower than what they could be expected to.
I regularly see programmers that come from a front end or "full stack" background try to solve data engineering problems, and end up with solutions that are 10x, 100x or sometimes 1000x slower than what they could be expected to.
No kidding.
It's not rare for me to catch Pull Requests with horrible performance holes. Like 'SELECT * FROM...' executed inside a loop where a single 'SELECT id FROM...' outside the loop could solve the problem.
It's not rare for me to catch Pull Requests with horrible performance holes. Like 'SELECT * FROM...' executed inside a loop where a single 'SELECT id FROM...' outside the loop could solve the problem.
> either god-tier ninja programmers, or rose-tinted specs
False dichotomy.
Removing some of the many, many layers of crud that hobbles our incredibly powerful machines does not take ninja-skills. It does however take the knowledge of what is possible, which is why it is often older developers who bring it up.
Tools that "manage" complexity very often beget more complexity that begets more tools that beget more complexity. Because each layer always turns out to be somewhat less good (and more leaky) at managing complexity than hoped for, and always turns out to add more complexity itself than initially anticipated.
This is not a new trend. We had "middleware abstraction layers" in the 90s. That kept me amused for longer than I care to admit.
So instead of trying to "manage" complexity, and usually failing, reduce complexity where you can. And as I wrote, it actually isn't that hard once you convince yourself that it's actually possible.
"You'll be amazed"
False dichotomy.
Removing some of the many, many layers of crud that hobbles our incredibly powerful machines does not take ninja-skills. It does however take the knowledge of what is possible, which is why it is often older developers who bring it up.
Tools that "manage" complexity very often beget more complexity that begets more tools that beget more complexity. Because each layer always turns out to be somewhat less good (and more leaky) at managing complexity than hoped for, and always turns out to add more complexity itself than initially anticipated.
This is not a new trend. We had "middleware abstraction layers" in the 90s. That kept me amused for longer than I care to admit.
So instead of trying to "manage" complexity, and usually failing, reduce complexity where you can. And as I wrote, it actually isn't that hard once you convince yourself that it's actually possible.
"You'll be amazed"
Say you're starting a new project, how would you keep the complexity down? It's a hard subject and I'll admit I'm prone to follow the interesting tech.
Use TDD and hexagonal architecture.
That is, start with pure model and build outward from there. Discipline yourself to build only what you really need using TDD: only write production code for a failing test case. Only write enough production code to just barely make the test pass. This will lead to silly code. Refactor this to sensible code when all the tests are green and you sense duplication.
Avoid starting with environmental things: infrastructure, frameworks, databases, etc.
Don't mock.
That is, start with pure model and build outward from there. Discipline yourself to build only what you really need using TDD: only write production code for a failing test case. Only write enough production code to just barely make the test pass. This will lead to silly code. Refactor this to sensible code when all the tests are green and you sense duplication.
Avoid starting with environmental things: infrastructure, frameworks, databases, etc.
Don't mock.
My graying computer organization professor always referred to the days of tech yore as “the bad old days.” I think that is something we should consider doing more. Things look simple in retrospect but in the midst of it, it is often complicated from day to day.
> We use docker for almost everything, and it’s a lifesaver.
While we're sharing anecdotes, we are using Docker for a part of our CI pipeline, and it's the only thing that constantly breaks for some reason.
While we're sharing anecdotes, we are using Docker for a part of our CI pipeline, and it's the only thing that constantly breaks for some reason.
My team as well has fixed a lot of flaky tests by reducing the number of places we use docker. It's the most finicky part of our infrastructure. I always wonder what it would be like if comments were tagged with the technologies in use by the commenter; would we find that the strongest proponents of docker are working in stacks that have a poorer isolation/deployment story? I remember trying to package Ruby apps and thinking "huh, this is a pain." If that was the world I was in, I can see why shipping an image of my OS is a lifesaver.
Yes, that's a good point. I have two Rails sites (side projects) and I've never worked with something so finicky. Something always breaks, and after every OS update I habe troubles getting my dev environment back up. I can see the appeal for Docker there.
But with all my other stuff this is really not an issue. I have a few small services written in Python, and some old stuff in PHP, and that stuff is so easy to run and deploy on pretty much any OS, and adding Docker would just add pointless complexity.
But with all my other stuff this is really not an issue. I have a few small services written in Python, and some old stuff in PHP, and that stuff is so easy to run and deploy on pretty much any OS, and adding Docker would just add pointless complexity.
[deleted]
I've wondered a lot about what the value prop of Docker is for simple Go applications and agree that if you can get away with it, ignoring Docker can be a good way to go.
One thing a Docker container gets you for a simple Go app is the ability to run it on any platform that runs container images (like ECS); that's more flexibility than you get with something that needs a proper VPS to run. Everything knows how to run a Docker image.
But if you don't care about stuff like ECS, and you're not trying to harden or isolate your runtime, and you don't do anything funky with networking --- this describes lots of apps --- sure!
One thing a Docker container gets you for a simple Go app is the ability to run it on any platform that runs container images (like ECS); that's more flexibility than you get with something that needs a proper VPS to run. Everything knows how to run a Docker image.
But if you don't care about stuff like ECS, and you're not trying to harden or isolate your runtime, and you don't do anything funky with networking --- this describes lots of apps --- sure!
For simple applications Docker doesn't provide much value at all. Unless of cause you have multiple of these simple applications, and they each have different requirements, but aren't large enough to efficiently utilize an entire VM.
The Go applications that can be distributed as a single binary, SpringBoot application, or a C program packaged up as a static binary or in a .deb/.rpm package, those application may not benefit much from Docker.
Where Docker does help is when it comes to distributing applications written in languages that honestly don't have great ways of distributing them. As much as I love Python, deploying to production can be a little tricky. Docker allows you to know exactly what is currently in production.
For many developers, and operations people, the thing they really want is the orchestration tools Docker/containerization brings. Even if it's just Docker-compose, you're now easily able to recreate entire stacks in different environments. That's providing real value for almost all cases.
The Go applications that can be distributed as a single binary, SpringBoot application, or a C program packaged up as a static binary or in a .deb/.rpm package, those application may not benefit much from Docker.
Where Docker does help is when it comes to distributing applications written in languages that honestly don't have great ways of distributing them. As much as I love Python, deploying to production can be a little tricky. Docker allows you to know exactly what is currently in production.
For many developers, and operations people, the thing they really want is the orchestration tools Docker/containerization brings. Even if it's just Docker-compose, you're now easily able to recreate entire stacks in different environments. That's providing real value for almost all cases.
> One thing a Docker container gets you for a simple Go app is the ability to run it on any platform that runs container images (like ECS); that's more flexibility than you get with something that needs a proper VPS to run. Everything knows how to run a Docker image.
It is kind of bonkers that we've gotten to a stage where more things know how to run a container than know how to run a statically linked binary.
It is kind of bonkers that we've gotten to a stage where more things know how to run a container than know how to run a statically linked binary.
"Running a binary" is still super easy. All you need to do is take account of log management, process isolation, networking configs (port forwarding), service discovery, health checking, data storage folders, fault-tolerant rollbacks, binary placement (putting the x86 binary on the x86 machine & arm on the arm machine), config files, environment variables, the working directory the process needs, etc.
Anything that "runs a binary somewhere" will essentially become docker given enough time. Docker is just "run this binary somewhere" everything above packaged into it.
Anything that "runs a binary somewhere" will essentially become docker given enough time. Docker is just "run this binary somewhere" everything above packaged into it.
> Anything that "runs a binary somewhere" will essentially become docker given enough time.
Docker... doesn't do nearly any of those things?
Docker... doesn't do nearly any of those things?
Docker attempts to do all of these things. It may not do them well but it provides a framework/opinion on how this should all be managed:
1. binary + configs (PWD, files, ENV): baked into the image's Dockerfile
2. Secrets: passed in via env vars at runtime
3. Service discovery: docker-compose exposes container name as a hostname and provides a DNS record for this available in all netns that are connected to the same network.
4. data storage folders: Volumes https://docs.docker.com/storage/volumes/#share-data-among-ma...
5. Health checking: https://docs.docker.com/engine/reference/builder/#healthchec...
6. binary placement: https://www.docker.com/blog/multi-arch-build-and-images-the-...
7. networking configs: https://docs.docker.com/network/
8. log management: to file or to syslog over TCP (super easy to ship to ELK) https://docs.docker.com/config/containers/logging/configure/
9. fault-tolerant rollbacks: if you tag images correctly and you are upgrading from v1 -> v2 you can just replace the tag with :v1 to rollback to the correct version which will also be cached on the machine. If your registry is down you can still rollback.
10. Process isolation: this is exactly what docker is. Process isolation implemented in the kernel (sometimes).
1. binary + configs (PWD, files, ENV): baked into the image's Dockerfile
2. Secrets: passed in via env vars at runtime
3. Service discovery: docker-compose exposes container name as a hostname and provides a DNS record for this available in all netns that are connected to the same network.
4. data storage folders: Volumes https://docs.docker.com/storage/volumes/#share-data-among-ma...
5. Health checking: https://docs.docker.com/engine/reference/builder/#healthchec...
6. binary placement: https://www.docker.com/blog/multi-arch-build-and-images-the-...
7. networking configs: https://docs.docker.com/network/
8. log management: to file or to syslog over TCP (super easy to ship to ELK) https://docs.docker.com/config/containers/logging/configure/
9. fault-tolerant rollbacks: if you tag images correctly and you are upgrading from v1 -> v2 you can just replace the tag with :v1 to rollback to the correct version which will also be cached on the machine. If your registry is down you can still rollback.
10. Process isolation: this is exactly what docker is. Process isolation implemented in the kernel (sometimes).
Docker doesn't have any infrastructure for managing configs. Nor secrets. It doesn't have log shippers or storage -- and "to file or syslog over TCP" are definitely not the recommended ways! The only thing it really "does" there is process isolation.
If everything compiled to a statically linked binary, sure. But most things don't compile to static binaries, and platforms are built to run things in all kinds of languages.
Meanwhile: Docker can be pretty insane as a runtime (we don't use it for that!) but it's not an insane packaging format.
Meanwhile: Docker can be pretty insane as a runtime (we don't use it for that!) but it's not an insane packaging format.
[deleted]
Well, turning a static binary into a docker image is basically just adding some metadata on how to run it. Whereas turning an image into a static binary is much harder, so it makes sense the world standarized on the more flexible format.
This was exactly my thought. Sure you don't need Docker. But if you want to effortlessly ship your application to a number of cloud platforms where you don't need to concern yourself with the host-level security aspects, it's hard to beat a Docker image.
> This might end up getting a lot of hate.
Why should I care whether you choose to use Docker (or other piece of software)?
If your use-case doesn't require it, then you probably shouldn't use it. I do take issue with your evidence however:
> Deploying the app is as easy as: > scp app user@host: > ssh user@host “nohup ./app”
Not if you're deploying to production it isn't, unless this is Hello World. I've conceived quite a few from-scratch deployments for Go applications. Yes, it's simple, but there are a lot of complicating factors in an application of any substantial complexity:
- ENV files
- credentials for deployment
- database migration
- supporting files (eg. web assets)
- something to actually make sure your application stays up (eg. systemd)
Those are the first things that come to mind, and overall it's probably a lot less complex deployment than stacks other than Go allow. Let me tell you why I do use Docker - for dev and deployment, but not production:
- Reproducible builds for all apps on the correct version of Golang
- No dependency issues among members of the dev team. Everything you need is in the docker images.
- Easy imitation of production stack (again, correct versions, etc)
Those are the main reasons. I think the most important part is to make sure everybody is building the same application. Here is an example:
Suppose my colleague is using Debian and installed Go with apt. He's stuck on 1.13, but I'm using Arch and got 1.15 as soon as it came out. Now our builds are different. This is why we use docker to run applications in dev, and build for production. Or suppose a colleague is using Windows or Mac to build. There are still differences! Using Docker means everyone is building in roughly the same environment.
I realize certainly that some organizations control more tightly what OS an employee is using for this sort of thing. Mine doesn't, and Docker helps to keep us all on the same page (the Linux page, of course).
Why should I care whether you choose to use Docker (or other piece of software)?
If your use-case doesn't require it, then you probably shouldn't use it. I do take issue with your evidence however:
> Deploying the app is as easy as: > scp app user@host: > ssh user@host “nohup ./app”
Not if you're deploying to production it isn't, unless this is Hello World. I've conceived quite a few from-scratch deployments for Go applications. Yes, it's simple, but there are a lot of complicating factors in an application of any substantial complexity:
- ENV files
- credentials for deployment
- database migration
- supporting files (eg. web assets)
- something to actually make sure your application stays up (eg. systemd)
Those are the first things that come to mind, and overall it's probably a lot less complex deployment than stacks other than Go allow. Let me tell you why I do use Docker - for dev and deployment, but not production:
- Reproducible builds for all apps on the correct version of Golang
- No dependency issues among members of the dev team. Everything you need is in the docker images.
- Easy imitation of production stack (again, correct versions, etc)
Those are the main reasons. I think the most important part is to make sure everybody is building the same application. Here is an example:
Suppose my colleague is using Debian and installed Go with apt. He's stuck on 1.13, but I'm using Arch and got 1.15 as soon as it came out. Now our builds are different. This is why we use docker to run applications in dev, and build for production. Or suppose a colleague is using Windows or Mac to build. There are still differences! Using Docker means everyone is building in roughly the same environment.
I realize certainly that some organizations control more tightly what OS an employee is using for this sort of thing. Mine doesn't, and Docker helps to keep us all on the same page (the Linux page, of course).
How is it configured? where do the log files go? what ports is it exposing? how is it health checked?
As much as anything else Docker (+/- compose et al) is a default set of answers to all these type of questions where everybody knows without even having to ask either the answers or where to find them. And its great that for about half of them they look identical to developers and devops / ops staff.
As much as anything else Docker (+/- compose et al) is a default set of answers to all these type of questions where everybody knows without even having to ask either the answers or where to find them. And its great that for about half of them they look identical to developers and devops / ops staff.
That will be asked and fixed by the first sysadmin/devops person they might eventually hire.
Most people don't seem to understand how simple Docker actually is when used on the correct platform. On Linux, it's not using a VM, it's running almost on bare metal. The overhead is ~1%.
For that ~1%, you get a build/development/production environment that is cacheable by layers, and identical wherever you run it.
For that ~1%, you get a build/development/production environment that is cacheable by layers, and identical wherever you run it.
None of which is needed for statically compiled go binaries
Its not needed, but its sure helpful. Docker handles much more than just app distribution.
Health checking, service recovery, process isolation, networking, log management, etc.
Sure, everything is possible without using Docker, but why not use an industry standard tool that does a lot of the heavy lifting, especially when it comes with as little overhead as docker?
Health checking, service recovery, process isolation, networking, log management, etc.
Sure, everything is possible without using Docker, but why not use an industry standard tool that does a lot of the heavy lifting, especially when it comes with as little overhead as docker?
until you start doing stuff to disk, then everything starts getting hard.
At a previous job, our build pipeline
* Built the app (into a self contained .jar, it was a JVM shop)
* Put the app into a Ubuntu Docker image. This step was arguably unnecessary, but the same way Maven is used to isolate JVM dependencies ("it works on my machine"), the purpose of the Docker image was to isolate dependencies on the OS environment.
* Put the Docker image onto an AWS .ami that only had Docker on it, and the sole purpose of which was to run the Docker image.
* Combined the AWS .ami with an appropriately sized EC2.
* Spun up the EC2s and flipped the AWS ELBs to point to the new ones, blue green style.
The beauty of this was the stupidly simple process and complete isolation of all the apps. No cluster that ran multiple diverse CPU and memory requirement apps simultaneously. No K8s complexity. Still had all the horizontal scaling benefits etc.
* Built the app (into a self contained .jar, it was a JVM shop)
* Put the app into a Ubuntu Docker image. This step was arguably unnecessary, but the same way Maven is used to isolate JVM dependencies ("it works on my machine"), the purpose of the Docker image was to isolate dependencies on the OS environment.
* Put the Docker image onto an AWS .ami that only had Docker on it, and the sole purpose of which was to run the Docker image.
* Combined the AWS .ami with an appropriately sized EC2.
* Spun up the EC2s and flipped the AWS ELBs to point to the new ones, blue green style.
The beauty of this was the stupidly simple process and complete isolation of all the apps. No cluster that ran multiple diverse CPU and memory requirement apps simultaneously. No K8s complexity. Still had all the horizontal scaling benefits etc.
I feel like if you have an AMI you don't need docker. If you have docker you don't need a new AMI each time.
For me it's about knowing "what is running." If I can get a binary, docker image, or AMI that lets me know exactly what is running then that's all I need to really care about. For docker without Fargate, k8s, nomad, or etc. it's probably best to simply have an ansible or salt config which pulls down the binary/docker image and updates a systemd service or similar on server group "a" and then you do the same thing to server group "b".
Occasionally you create a new base AMI with updates and expected scripts/agents/etc. and replace the existing instances.
[edit: typo]
For me it's about knowing "what is running." If I can get a binary, docker image, or AMI that lets me know exactly what is running then that's all I need to really care about. For docker without Fargate, k8s, nomad, or etc. it's probably best to simply have an ansible or salt config which pulls down the binary/docker image and updates a systemd service or similar on server group "a" and then you do the same thing to server group "b".
Occasionally you create a new base AMI with updates and expected scripts/agents/etc. and replace the existing instances.
[edit: typo]
Yeah the Docker step isn't strictly speaking necessary for the deployment, it's more for devs with diverse OS's to be able to isolate dependencies on the OS. At JVM shops, the same .jar unfortunately doesn't reliably and reproducibly run the same way across machines. The JVM can only isolate dependencies within itself - if your app requires eg certain /etc/host values, that's something that can and probably should go into a Docker file that isolates dependencies on that, and make that explicit.
As for the AMIs the benefit of always making new ones and storing them (tagging them with the git commit hash) vs mutating a running ec2 is it makes it incredibly simple to roll back - if a release goes wrong, just spin up new EC2s using the previous AMI and viola.
In general we prefer immutability of all things. State is the devil, especially when it comes to "what's running in prod?"
As for the AMIs the benefit of always making new ones and storing them (tagging them with the git commit hash) vs mutating a running ec2 is it makes it incredibly simple to roll back - if a release goes wrong, just spin up new EC2s using the previous AMI and viola.
In general we prefer immutability of all things. State is the devil, especially when it comes to "what's running in prod?"
Yes. I do k8s all day and this solution wins every time for simplicity where possible. I get it -- service discovery, healthchecks, no downtime deployments, the rest of the k8s goodies -- doing all that in a generic way is hard!
So just orchestrate your business logic. There's no shame in that. And you might well come away with a cleaner system than if you tried to wedge everything into Kubernetes or some other orchestrator.
I've come full circle -- started with Mesos, moved to Kubernetes, dabbled with Nomad/Consul. I love what Kubernetes offers. Avoid the complexity if you can. If you don't NEED to pay the price. Just don't.
So just orchestrate your business logic. There's no shame in that. And you might well come away with a cleaner system than if you tried to wedge everything into Kubernetes or some other orchestrator.
I've come full circle -- started with Mesos, moved to Kubernetes, dabbled with Nomad/Consul. I love what Kubernetes offers. Avoid the complexity if you can. If you don't NEED to pay the price. Just don't.
It’s just funny to use three separate encapsulating technologies inside each other... when do we add another layer?
When it provides benefits which either can't be solved by or clearly don't belong to any other layer.
[deleted]
Do you have a database, a queue service, a caching layer (or 3), a session store? If all your app needs is an http server then sure, why bother with docker... But if it's more complicated than that, it's great to have standard recipes for alignment in local development, a problem docker solves very well
These things can also be solved by using distro packages and `cp`-ing config files. Docker introduces a ton of complexity into the stack.
Using Docker to solve these problems is like hiring a construction company to build a birdhouse. Sure, it'll save you the work of having to build the birdhouse yourself, but...it's major overkill, and could end up costing you more effort than other options.
Now, if you were running a complex suite of mircoservices, then there might be a good reason to use containers. But containers are like JavaScript; you shouldn't jump for them before you've weighed your other options.
Using Docker to solve these problems is like hiring a construction company to build a birdhouse. Sure, it'll save you the work of having to build the birdhouse yourself, but...it's major overkill, and could end up costing you more effort than other options.
Now, if you were running a complex suite of mircoservices, then there might be a good reason to use containers. But containers are like JavaScript; you shouldn't jump for them before you've weighed your other options.
Reproducibility with a well-written Docker image is well worth it over "cp-ing config files" shiver.
Docker isn't Kubernetes -- it's not that complicated.
Docker isn't Kubernetes -- it's not that complicated.
Docker images aren’t built in a reproducible way unless you use something like Nix to create them
cp-ing config files is typically done by using config management like puppet, chef, ansible, or saltstack via templating based on server type, location, etc.
What's so risky about cp-ing config files as opposed to typing "COPY" or mounting a volume in a Dockerfile?
If you use a distro like Debian you shouldn't be worried about breaking changes, especially if you're subscribed to updates/relnotes for the critical parts of your stack (e.g., an announcements mailing list).
Those are good practices to have regardless of your platform.
If you use a distro like Debian you shouldn't be worried about breaking changes, especially if you're subscribed to updates/relnotes for the critical parts of your stack (e.g., an announcements mailing list).
Those are good practices to have regardless of your platform.
You don’t need complex microservices do get some benefits from reproducible production environments. If you have multiple machines and need all of them to have an identical environment, there is no better way. Everything else is ridiculously error prone and complicated. Ever tried to debug why two Java installations have different classpaths? Good luck. Need to upgrade different system level dependencies but your applications needs a specified version? How will you do that?
Docker is like the static typing of deployments. It may seem a bit over the top at first, but it will really make your life easier.
Docker is like the static typing of deployments. It may seem a bit over the top at first, but it will really make your life easier.
The Nix package manager is great for this - the package language was initially a bit more foreign to me than Dockerfile, but the package manager gives the same reproduceability benefits, integrates well with the OS, and being purely functional/immutable let’s you roll the system state back at any time, like Git
Docker is just a layer on top of cgroups, and Dockerfiles are basically just shell scripts. It doesn't have to be a huge undertaking
Docker and its tightly coupled sibling programs amount to millions of lines of code. It took billions of dollars in funding to make. There's a reason it's classified as a PaaS.
It is far more than a "layer on top of cgroups".
If you believe "Dockerfiles are basically shell scripts", you could try using shell scripts.
It is far more than a "layer on top of cgroups".
If you believe "Dockerfiles are basically shell scripts", you could try using shell scripts.
Already done https://github.com/p8952/bocker/blob/master/README.md (Docker implemented in around 100 lines of bash.)
I have most of those things. I just use hosted versions of those things.
Even locally?
In my case, I run all services locally without docker. apt-get seems to work just fine.
Seems a bit brittle.
Oddly, most of this brittle nature helps build a more robust product.
That is, yes, it will break often in small ways. However, most of the breaks will be from version upgrades or machine changes. The sooner you find those, the better.
That is, yes, it will break often in small ways. However, most of the breaks will be from version upgrades or machine changes. The sooner you find those, the better.
Honestly, it hasn't been. Probably helps that I run Linux, but would probably do the same on Mac. Managing that extra virtualization layer has always been more pain than just relying on services that are just always there.
Most issues are because of changes between software versions. Some very few are because of environment, but that's what a staging environment is for.
We just had a hosted development database, development redis, etc. It works just fine.
> database, a queue service, ... a session store
I use these three both remote and locally, and start all of these using a simple shell script.
I use these three both remote and locally, and start all of these using a simple shell script.
I think you're pointing out the double-edged sword that is Docker; it's very easy to run and experiment with new technologies, even ones you might not be organizationally ready for. The "local" kafka that you used for a proof of concept is now AWS MSK in production.
Hey, I too don't want to use Docker, but I find no alternative.
I use docker-compose to get these running: memcache, MySQL, NGINX and postfix
I tried other things: - Ansible: very slow and requires lot of config files - Bash scripts: requires lot of build steps in a language I don't really like - Fabric: good as the scripts are more pythonic, but still requires manual installation of all services
The good thing about Docker is its caching mechanism. It builds the containers only if there are any changes. What are other (containerless) solutions for that?
I use docker-compose to get these running: memcache, MySQL, NGINX and postfix
I tried other things: - Ansible: very slow and requires lot of config files - Bash scripts: requires lot of build steps in a language I don't really like - Fabric: good as the scripts are more pythonic, but still requires manual installation of all services
The good thing about Docker is its caching mechanism. It builds the containers only if there are any changes. What are other (containerless) solutions for that?
Postfix is really hard to run in container. Add rspamd, opendkim, opendmarc and I would just run them in their dedicated vm then.
Panubo has a pretty good postfix images for Docker. It handles most of the stuff:
https://hub.docker.com/r/panubo/postfix/
Source: https://github.com/panubo/docker-postfix
https://hub.docker.com/r/panubo/postfix/
Source: https://github.com/panubo/docker-postfix
You accept that these are benefits, so why don't you want to use Docker?
Running where though? Locally? If not, why use Memcahed or Nginx?
Nah, running on production.
For local I don't use Docker.
For local I don't use Docker.
But there’s almost no point in running Memcached and Nginx on the same machine.
Docker isn't as hard to learn as the author makes it out to be. You can learn it in less than 5 hours (take a 10 hour course at 2x the speed). Golang takes significantly longer to learn. This may sound obvious, but the speed of learning depends on what you already know.
A docker container is instantiated from a docker image, it's similar to object oriented programming: an object is instantiated from a class. Want to manipulate containers? Use 'docker container ...'. Want to manipulate docker images? Use 'docker image ...'.
There's also a differentiation from production vs. development environments. How do you onboard new developers? If you use docker, you can put a docker-compose.yml file in the repo and when you onboard a new developer, it's as simple as 'git clone ${the_repo} && docker-compose up'. If you want to use docker in production, you're probably using Kubernetes, and I would agree that there's an increased cost in complexity. Want to scale the docker containers across multiple servers? The logical choice is to use Kubernetes to orchestrate the containers. If you're scaling vertically, sure, use your current method. Golang is fast.
As your company grows & scales, I think you'll run into deployment issues using scp for deployments. What happens if two developers deploy at the same time? Ideally you put every deployment through a single "pipe" where if two people deploy at the same time, one deployment/testing process would run after the other in a serial manner, causing the second deployment to fail if there were a conflict.
Arguing about tech turns into a religious debate at a certain point. Use docker/Kubernetes if you need to, and if you can get by without using them, don't use them. Docker is awesome for solving versioning problems and onboarding new employees. Kubernetes is awesome at scaling and deployments. If your employees all agree to use the same version of software, there's no need to use docker or Kubernetes.
But hey, at least docker/Kubernetes give you the choice to think freely and if you see some cool library that you want to use written in some obscure language or version, it's easy peasy.
A docker container is instantiated from a docker image, it's similar to object oriented programming: an object is instantiated from a class. Want to manipulate containers? Use 'docker container ...'. Want to manipulate docker images? Use 'docker image ...'.
There's also a differentiation from production vs. development environments. How do you onboard new developers? If you use docker, you can put a docker-compose.yml file in the repo and when you onboard a new developer, it's as simple as 'git clone ${the_repo} && docker-compose up'. If you want to use docker in production, you're probably using Kubernetes, and I would agree that there's an increased cost in complexity. Want to scale the docker containers across multiple servers? The logical choice is to use Kubernetes to orchestrate the containers. If you're scaling vertically, sure, use your current method. Golang is fast.
As your company grows & scales, I think you'll run into deployment issues using scp for deployments. What happens if two developers deploy at the same time? Ideally you put every deployment through a single "pipe" where if two people deploy at the same time, one deployment/testing process would run after the other in a serial manner, causing the second deployment to fail if there were a conflict.
Arguing about tech turns into a religious debate at a certain point. Use docker/Kubernetes if you need to, and if you can get by without using them, don't use them. Docker is awesome for solving versioning problems and onboarding new employees. Kubernetes is awesome at scaling and deployments. If your employees all agree to use the same version of software, there's no need to use docker or Kubernetes.
But hey, at least docker/Kubernetes give you the choice to think freely and if you see some cool library that you want to use written in some obscure language or version, it's easy peasy.
This is an oversimplification and Docker is easy from a basic usage standpoint like most tutorials are. The hard part is when you start configuring and configuring to your own requirements. What baseimage should you use? How about logging? How about PID 1 problem? Do I need an init system? How about SSH? Remote Docker containers cannot be accessed remotely for ex AWS Fargate. How about restarting servers? I need to do it gracefully. How about migrations? One-off scripts?
Personally, I love using Docker but it’s those small things that make it hard and all the complexity around setting them up.
Personally, I love using Docker but it’s those small things that make it hard and all the complexity around setting them up.
Just replying to your comment, not trying to start an argument but just throwing out some thoughts. Please let me know if I'm off track in my response.
What baseimage should you use?
- You can always create your own docker base images. I do agree that it gets confusing when you pull a docker image and it's not using the linux flavor that you were expecting. Building a docker image is quite simple though.
How about logging?
- If using Kubernetes, you can use a 'side car' pattern with a log exporter.
How about PID 1 problem?
- I'm a little confused what you mean, but I think you are referring to killing PID 1, which would kill the docker container. If you're using Kubernetes in production, then Kubernetes would solve the problem of routing to docker containers that are up or down.
How about SSH?
- You would have to SSH into the host with the docker container present, and drop into the docker container thereafter.
Remote Docker containers cannot be accessed remotely for ex AWS Fargate.
- I haven't used AWS Fargate before, no comment.
How about restarting servers?
- Handled by Kubernetes, totally ok to restart the servers and requests won't be routed to dead (5xx) containers.
How about migrations?
- Deployments are simple using Kubernetes (built in rolling deployment, but also easy to do blue/green deployments). If you want to migrate a database, then it'd be the same process as non-container deployments.
What baseimage should you use?
- You can always create your own docker base images. I do agree that it gets confusing when you pull a docker image and it's not using the linux flavor that you were expecting. Building a docker image is quite simple though.
How about logging?
- If using Kubernetes, you can use a 'side car' pattern with a log exporter.
How about PID 1 problem?
- I'm a little confused what you mean, but I think you are referring to killing PID 1, which would kill the docker container. If you're using Kubernetes in production, then Kubernetes would solve the problem of routing to docker containers that are up or down.
How about SSH?
- You would have to SSH into the host with the docker container present, and drop into the docker container thereafter.
Remote Docker containers cannot be accessed remotely for ex AWS Fargate.
- I haven't used AWS Fargate before, no comment.
How about restarting servers?
- Handled by Kubernetes, totally ok to restart the servers and requests won't be routed to dead (5xx) containers.
How about migrations?
- Deployments are simple using Kubernetes (built in rolling deployment, but also easy to do blue/green deployments). If you want to migrate a database, then it'd be the same process as non-container deployments.
"Docker is easy from a basic usage standpoint like most tutorials are"
This applies to basically any software or tooling once you get away from the initial tutorials. They wouldn't be tutorials if they covered all sorts of specific use cases particular to your own situation.
"The hard part is when you start configuring and configuring to your own requirements. What baseimage should you use? How about logging?"
Either the same distro you would use on a bespoke server, or you can optimize with alpine or scratch. Logging can be as simple or complicated as you want it to be, regardless if you use docker or not.
"How about PID 1 problem?"
We use s6-overlay [0] to manage this. When a service dies, a script gets executed, giving us control over how to handle it.
"How about SSH?"
For us, we use ansible to provision/set up kubernetes with ssh keys, then access pods and containers through kubernetes' cli. It isn't particularly -nice-, but it does work.
"How about restarting servers? How about migrations? One-off scripts?"
This is managed through kubernetes. You can restart servers and make deployments.
[0]: https://github.com/just-containers/s6-overlay#features
This applies to basically any software or tooling once you get away from the initial tutorials. They wouldn't be tutorials if they covered all sorts of specific use cases particular to your own situation.
"The hard part is when you start configuring and configuring to your own requirements. What baseimage should you use? How about logging?"
Either the same distro you would use on a bespoke server, or you can optimize with alpine or scratch. Logging can be as simple or complicated as you want it to be, regardless if you use docker or not.
"How about PID 1 problem?"
We use s6-overlay [0] to manage this. When a service dies, a script gets executed, giving us control over how to handle it.
"How about SSH?"
For us, we use ansible to provision/set up kubernetes with ssh keys, then access pods and containers through kubernetes' cli. It isn't particularly -nice-, but it does work.
"How about restarting servers? How about migrations? One-off scripts?"
This is managed through kubernetes. You can restart servers and make deployments.
[0]: https://github.com/just-containers/s6-overlay#features
It’s also worth mentioning that docker is a leaky abstraction. Docker employs tricks to create a perception of containerization. This means that if any steps are missed or forgotten your system could become unusable.
Examples:
- something failed and the app is partially up? Did it take out your ssh port? Good luck getting access to fix the issue
- app fails on boot? good luck kicking the tires interactively. best trick I’ve found is to tail /dev/null upon startup. Aside from that it’s a fine tool. I just don’t need it. The only reason I do is because a technologist who came before me thought I did.
I find that we live in a world where abstractions between me and bare metal keep multiplying and I frankly don’t like it. Maybe I’m just an old fart now.
This leads me to asking a question I constantly ponder. Isn’t the need for yet another abstraction clear evidence that the lower abstraction is weak? I understand that abstractions of varying levels are useful in different situations. These thoughts are specifically pointed at helm, k8s and docker.
Examples:
- something failed and the app is partially up? Did it take out your ssh port? Good luck getting access to fix the issue
- app fails on boot? good luck kicking the tires interactively. best trick I’ve found is to tail /dev/null upon startup. Aside from that it’s a fine tool. I just don’t need it. The only reason I do is because a technologist who came before me thought I did.
I find that we live in a world where abstractions between me and bare metal keep multiplying and I frankly don’t like it. Maybe I’m just an old fart now.
This leads me to asking a question I constantly ponder. Isn’t the need for yet another abstraction clear evidence that the lower abstraction is weak? I understand that abstractions of varying levels are useful in different situations. These thoughts are specifically pointed at helm, k8s and docker.
Regarding your final remark, almost all abstraction is the result of someone being impatient and fixing the problem of the decade. Why does anyone need Python for anything if C++ is so much faster at runtime? Because C++ takes more time to learn, code, debug, and fix bugs. These abstractions aren't weak as a whole, but they do have their downsides, so this pain endured encourages bright developers to create another layer of abstraction that avoids exposing these same rough edges to new developers.
> - something failed and the app is partially up? Did it take out your ssh port? Good luck getting access to fix the issue
Not sure how it could ever knock out your SSH port unless you run a container like
> - something failed and the app is partially up? Did it take out your ssh port? Good luck getting access to fix the issue
Not sure how it could ever knock out your SSH port unless you run a container like
ports:
22:22
Wondering if OP is running Sshd inside his containers and forwarding ports - effectively using them as a VM
> Maybe I’m just an old fart now.
As an old fart I sympathise greatly with this, but objectively things have always been more and more abstracted as time goes on, and old farts are always resistant because it’s they way they always did it.
I was young once, came in and asked “why all these old expensive sun machines when we could just run linux”. I got across VMs eventually, but things like docker was against the natural order of things, and seemed to just be a crutch to me. Apt is a crunch to those who don’t want to deal with missing dependencies. Debs are there to avoid configure/make/make install issues. Why even have a make file, just handcraft your gcc lines directly.
An older colleague used to run an internal web server running on irix with a cgis written in C, he moved on to php and linux, then to node and docker. He moved with the time, and I respect that.
I doubt you wrote your programs in assembly throughout your career, so there’s always been abstraction. There are times you need to hand write assembly, but they are rare. There are times you have to run on physical hardware, but they are rare. Why is an OS or a VM the perfect level of abstraction, and not docker?
Docker I can see has advantages in many cases. I need to deploy some new infrastructure to a small company, radius, syslog, dns, etc. Main and backup.
I could do it in the traditional way I’ve done it for 2 decades by simply installing them and configuring them like it’s 1999. And it would work and be great
Or I could use docker to do it, perhaps even use Andover to deploy. Why? Mainly to boost my resume, but I’m sure there are objective benefits to docker which I just can’t envisage at the moment because I’ve spent 2 decades doing things in the way that was new and cool in the days of Debian Potato, but now is old and presumably limited. Just like configure/make/make install was limited on Solaris, just like c based cgi was limited on irix.
As an old fart I sympathise greatly with this, but objectively things have always been more and more abstracted as time goes on, and old farts are always resistant because it’s they way they always did it.
I was young once, came in and asked “why all these old expensive sun machines when we could just run linux”. I got across VMs eventually, but things like docker was against the natural order of things, and seemed to just be a crutch to me. Apt is a crunch to those who don’t want to deal with missing dependencies. Debs are there to avoid configure/make/make install issues. Why even have a make file, just handcraft your gcc lines directly.
An older colleague used to run an internal web server running on irix with a cgis written in C, he moved on to php and linux, then to node and docker. He moved with the time, and I respect that.
I doubt you wrote your programs in assembly throughout your career, so there’s always been abstraction. There are times you need to hand write assembly, but they are rare. There are times you have to run on physical hardware, but they are rare. Why is an OS or a VM the perfect level of abstraction, and not docker?
Docker I can see has advantages in many cases. I need to deploy some new infrastructure to a small company, radius, syslog, dns, etc. Main and backup.
I could do it in the traditional way I’ve done it for 2 decades by simply installing them and configuring them like it’s 1999. And it would work and be great
Or I could use docker to do it, perhaps even use Andover to deploy. Why? Mainly to boost my resume, but I’m sure there are objective benefits to docker which I just can’t envisage at the moment because I’ve spent 2 decades doing things in the way that was new and cool in the days of Debian Potato, but now is old and presumably limited. Just like configure/make/make install was limited on Solaris, just like c based cgi was limited on irix.
Docker is not using chroot and iptables aren’t the most important part of “creating the perception”. The primary tools are Linux namespaces and cgroups. For many purposes, these abstractions aren’t too leaky (and sometimes quite useful without Docker, too)
Hmm this may have been because of the version of docker I used.
Updated the content
Updated the content
There's a throwaway line in there:
> Note, we did get a little more complicated and create a SystemD script to launch it at startup.
There's a good chance that most of what you want is in Systemd.
Restart a process if it fails health checks? Systemd. Do you want to isolate a process in a container? Systemd. Open privileged sockets? Systemd. You can do a lot with Systemd.
> Note, we did get a little more complicated and create a SystemD script to launch it at startup.
There's a good chance that most of what you want is in Systemd.
Restart a process if it fails health checks? Systemd. Do you want to isolate a process in a container? Systemd. Open privileged sockets? Systemd. You can do a lot with Systemd.
Lemme know when the catfight between systemd and docker is over.
I'm a bit busy with chroot and freebsd-jails over here.
I'm a bit busy with chroot and freebsd-jails over here.
We had great success using podman for a lot of systemd services
You don't have to worry about that if you don't use Docker as suggested in the OP.
systemd is great. I used it successfully for operations on many projects that I ran. It provides the standardized way to reliably run and manage services on a single machine. I also agree with opponents that shell was the de facto standard way to orchestrate things, allowing you to go from dev to prod without significant changes. I was a fan of runit as a process manager before systemd took over the space.
That said there is a problem when you want to run things on multiple machines. What then? Unix and systemd were not designed as an OS for machine clusters, even though it won't be hard to imagine the concepts to be stretched even to this realm.
The docker/kubernetes crowd has been targeting the multi-machine OS scene much sooner and better, even though the solution of let's pack all dependencies, the entire machine and then some to a fat package and then have it run by a scheduler complex as f seems not like the ultimate future of the space.
I took a look at Nomad as a replacement for Kubernetes, which is much much simpler and you can get up and running with a single binary within 2 hours using this tutorial [0]. Nomad supports not only docker as a backend, but you can run raw_exec backend as well - that is any binary, such as shell script, that you wish to run without packaging the entire world with it. Now imagine if only you could schedule to run systemd units using Nomad. You'd scale from a single machine OS to cluster OS just like that.
I predict that Kubernetes will NOT be the ultimate answer as many are hoping for, betting their careers on it. It is too complex and not supporting other than docker runtimes makes it much inferior product than Nomad. Simple solutions usually win ultimately, so Nomad or systemd extended for multi machine services or even just vanilla linux process management extended to cluster level seem like much saner approaches.
[0] https://medium.com/hashicorp-engineering/hashicorp-nomad-fro...
That said there is a problem when you want to run things on multiple machines. What then? Unix and systemd were not designed as an OS for machine clusters, even though it won't be hard to imagine the concepts to be stretched even to this realm.
The docker/kubernetes crowd has been targeting the multi-machine OS scene much sooner and better, even though the solution of let's pack all dependencies, the entire machine and then some to a fat package and then have it run by a scheduler complex as f seems not like the ultimate future of the space.
I took a look at Nomad as a replacement for Kubernetes, which is much much simpler and you can get up and running with a single binary within 2 hours using this tutorial [0]. Nomad supports not only docker as a backend, but you can run raw_exec backend as well - that is any binary, such as shell script, that you wish to run without packaging the entire world with it. Now imagine if only you could schedule to run systemd units using Nomad. You'd scale from a single machine OS to cluster OS just like that.
I predict that Kubernetes will NOT be the ultimate answer as many are hoping for, betting their careers on it. It is too complex and not supporting other than docker runtimes makes it much inferior product than Nomad. Simple solutions usually win ultimately, so Nomad or systemd extended for multi machine services or even just vanilla linux process management extended to cluster level seem like much saner approaches.
[0] https://medium.com/hashicorp-engineering/hashicorp-nomad-fro...
> Now imagine if only you could schedule to run systemd units using Nomad
With some tweaks to your unit files this is actually possible. Nomad has the concept of custom task drivers you can implement to make it schedule any kind of workload you like. I am maintaining a task driver which allows you to run systemd-nspawn containers with Nomad [1].
Using this task driver you can deploy your systemd units running inside a systemd-nspawn container into a Nomad cluster. If this sounds interesting to you I have written a how-to blog-post about this [2]
[1]: https://github.com/JanMa/nomad-driver-nspawn
[2]: https://janma.tk/2021-01-10/nomad-deploy-systemd-units/
With some tweaks to your unit files this is actually possible. Nomad has the concept of custom task drivers you can implement to make it schedule any kind of workload you like. I am maintaining a task driver which allows you to run systemd-nspawn containers with Nomad [1].
Using this task driver you can deploy your systemd units running inside a systemd-nspawn container into a Nomad cluster. If this sounds interesting to you I have written a how-to blog-post about this [2]
[1]: https://github.com/JanMa/nomad-driver-nspawn
[2]: https://janma.tk/2021-01-10/nomad-deploy-systemd-units/
When you say that Kubernetes only supports the Docker runtime I guess you meant only containers as it supports anything that implements CRI (Container Runtime Interface) like containerd.
Think I just saw from your article that Nomad supports straight up Java binaries but don't know how flexible that is
Think I just saw from your article that Nomad supports straight up Java binaries but don't know how flexible that is
What you describe is essentially the original CoreOS fleet[0] project. It's distributed systemd init files.
[0] https://github.com/coreos/fleet#fleet---a-distributed-init-s...
I find it ironic half of k8s mojo, etcd, came out of this project as well.
[0] https://github.com/coreos/fleet#fleet---a-distributed-init-s...
I find it ironic half of k8s mojo, etcd, came out of this project as well.
What would be the point of using Nomad to orchestrate systemd units instead of just using Nomad to raw_exec stuff? Nomad does health checks and all that, so systemd isn't really needed.
Btw i wrote an article about how great, easy and feature rich Nomad is which might interest you - https://atodorov.me/2021/02/27/why-you-should-take-a-look-at...
Btw i wrote an article about how great, easy and feature rich Nomad is which might interest you - https://atodorov.me/2021/02/27/why-you-should-take-a-look-at...
I've gotten incredibly far with packer-generated amis, a few build scripts and systemd to keep everything humming along. The only part I'm not a fan of is re-building amis due to the amount of time it takes to bake them.
I have a bunch of platforms that’re just jarfile monoliths (with external db/redis) that we deploy by provisioning the latest Amazon Linux onto EC2, then running a
Ansible to configure and add our local dependencies and the jarfile, then run the start script. While our Ansible step runs quickly enough, this is a nice way to not worry too much about managing our own AMIs.
(Two things really help here, we have Windows/macOS/Linux devs all developing and running the code locally so they’re pretty good at not accidentally becoming dependant on stuff outside the JVM, and the healthcheck endpoint the load balancers use is very comprehensive and over the years has been updated to include every non-detected failure or performance degradation.)
(Two things really help here, we have Windows/macOS/Linux devs all developing and running the code locally so they’re pretty good at not accidentally becoming dependant on stuff outside the JVM, and the healthcheck endpoint the load balancers use is very comprehensive and over the years has been updated to include every non-detected failure or performance degradation.)
Depends on how you bake them. NixOS plus an S3 bucket as a private nix cache should be pretty quick.
The only argument I have heard for Docker that makes any sense to me lately is that when your developers are using macOS and your servers run Linux, it’s nice to have dev/prod parity. Practically I think this speaks to either using better development environments, or writing code that is much more portable than what people try to get away with.
Eh I use docker as essentially disk images
Get something running in any environment really quickly because it is already a separate environment
Nobody has to deal with package managers messing up their system, or have to deal with package manager managers that lets them switch between different versions of a package manager just so they can run some obscure dependency before getting to the project
Get something running in any environment really quickly because it is already a separate environment
Nobody has to deal with package managers messing up their system, or have to deal with package manager managers that lets them switch between different versions of a package manager just so they can run some obscure dependency before getting to the project
You just deal with all the library and dependency stuff inside the Docker container instead :)
I was reading some systemd docs the other day, was stunned at how much it can do now
I'm not sure why systemD is needed for restarting a process what about a simple loop with an if statement? Process isolation to run a go binary on a presumably dedicated web server? Privileged sockets, not even sure what those might be.
The point of this post is to not complicate things, but what I hear is even more complicated be it via a docker alternative.
The point of this post is to not complicate things, but what I hear is even more complicated be it via a docker alternative.
> I'm not sure why systemD is needed for restarting a process what about a simple loop with an if statement?
It's not needed, but that's why a lot of older apps have their very custom solution to deal with that issue. (For example MySQL) Each with their custom exit condition handling, cleanup, backoff, etc. A simple loop also won't cleanup processes forked from the main server.
This 1kLoC script https://github.com/mysql/mysql-server/blob/8.0/scripts/mysql... can be almost entirely replaced with a few lines of systemd.service config. (with fairly logical names for each setting)
Now we can replace those with "Restart=always". If you want to be fancy, you could even add a watchdog.
> Process isolation to run a go binary on a presumably dedicated web server?
1. "presumably" 2. Apparently a web server which gets SSH sessions from devs - nice place to check for forwarded keys or collect passwords.
> Privileged sockets, not even sure what those might be.
Ports below 1024 (or whatever your local threshold is)
> but what I hear is even more complicated
It really isn't. Just unfamiliar to you.
It's not needed, but that's why a lot of older apps have their very custom solution to deal with that issue. (For example MySQL) Each with their custom exit condition handling, cleanup, backoff, etc. A simple loop also won't cleanup processes forked from the main server.
This 1kLoC script https://github.com/mysql/mysql-server/blob/8.0/scripts/mysql... can be almost entirely replaced with a few lines of systemd.service config. (with fairly logical names for each setting)
Now we can replace those with "Restart=always". If you want to be fancy, you could even add a watchdog.
> Process isolation to run a go binary on a presumably dedicated web server?
1. "presumably" 2. Apparently a web server which gets SSH sessions from devs - nice place to check for forwarded keys or collect passwords.
> Privileged sockets, not even sure what those might be.
Ports below 1024 (or whatever your local threshold is)
> but what I hear is even more complicated
It really isn't. Just unfamiliar to you.
if you go down the Restart=always route, please take a long look at StartLimitIntervalSec, RestartSec, StartLimitBurst. in some failure modes always isn't quite always always.
That's the whole point, you're not mysql. You don't need the sophistication of that magnitude. You need to run a standalone go binary to listen on a port. Simple. Why bother with the blackbox of systemD when you can perfectly get away with a basic shell script?
Yeah it really is complicated, probably just very familiar to you.
I Google systemD the first result has 160 sections just in the documentation table of contents alone [0].
[0] https://www.freedesktop.org/wiki/Software/systemd/
I Google systemD the first result has 160 sections just in the documentation table of contents alone [0].
[0] https://www.freedesktop.org/wiki/Software/systemd/
The linked page is not a documentation ToC. Why would you claim that?
For day to day usage there are two man pages which should be enough for reference: https://www.freedesktop.org/software/systemd/man/systemd.exe... and https://www.freedesktop.org/software/systemd/man/systemd.ser...
Or you could start with one of the blog posts linked in the page you referenced.
For day to day usage there are two man pages which should be enough for reference: https://www.freedesktop.org/software/systemd/man/systemd.exe... and https://www.freedesktop.org/software/systemd/man/systemd.ser...
Or you could start with one of the blog posts linked in the page you referenced.
The first link you sent is 1024 lines long, do you genuinely believe using something like this is preferred to a simple shell script + nohup? I'm amazed at the hoops some devs jump through just to get to work with their favorite tool.
That script is never going to be simple in a few years. You're going to keep adding features to it with time until it badly implementsa tiny fraction of what you already had available in your init system. The question could be: why spend time rewriting a broken restart script if you already have a good version available behind a config switch.
If you don't like how long that man page is, you should check the size of bash+nohub man page which you'd otherwise need :-)
If you don't like how long that man page is, you should check the size of bash+nohub man page which you'd otherwise need :-)
bash + nohup are elementary parts of using Linux, systemD is not and therefore learning it is added complexity.
By his peculiar capitalization you can tell he's a hater that still hasn't come to terms with it, 10 years later.
Or is tired of systemd being auto corrected to systems.
Using camel case means I'm a hater?
> I'm not sure why systemD is needed for restarting a process
You need a systemd unit file that starts your daemon properly when the server boots anyways (most common distros use systemd for init), and it's only a few more lines to make sure it gets restarted if it dies.
You need a systemd unit file that starts your daemon properly when the server boots anyways (most common distros use systemd for init), and it's only a few more lines to make sure it gets restarted if it dies.
Systemd config files are fairly simple and straightforward. It gives a simple interface to start/stop/restart processes and lets you do things like automatically start a process on server boot.
systemd unit files are cleaner than some init scripts, but they only have one implementation, which works on one libc and one kernel. uf.
> only have one implementation, which works on one libc and one kernel
I agree overall, but apparently systemd supports musl. No clue why or who uses it, but it's a thing.
I agree overall, but apparently systemd supports musl. No clue why or who uses it, but it's a thing.
Not according to several people who maintain patches outside of systemd explicitly for musl compatibility.
Did this change recently? Last I heard, systemd maintainers explicitly rejected attempts to support alternate libc implementations.
Did this change recently? Last I heard, systemd maintainers explicitly rejected attempts to support alternate libc implementations.
The perspective of the OP seems to stem from not having had to deal with the
problems docker is trying to address. Most importantly it ignores the problems
that arise on any dev team that is made up by more than one person.
One can do all the things that docker does without docker, but the knowledge needed to bring the basic features that docker does is far greater than learning how to use docker. In fact, if you know how to build something remotely close to docker, you'd be crazy not to use docker instead. At the very least, docker brings along industry standard conventions, and builds a common ground for everyone to work on without having to be onboarded on your home brewed packaging, distribution, storage, runtime and networking solution, regardless how simple it is.
On a side note, it seems like many trending topics on HN are heavily bashing new technologies (see k8s, rust, clouds, etc...) that are already widely adopted. Most communicate a resignation from learning something too complex. Docker is not complex, k8s is not complex, rust is not complex; the problems they are addressing are. Most teams developing things that run on the web will have to deal with these problems hopefully before they don't make it. The rest are wizards like the stackoverflow team.
One can do all the things that docker does without docker, but the knowledge needed to bring the basic features that docker does is far greater than learning how to use docker. In fact, if you know how to build something remotely close to docker, you'd be crazy not to use docker instead. At the very least, docker brings along industry standard conventions, and builds a common ground for everyone to work on without having to be onboarded on your home brewed packaging, distribution, storage, runtime and networking solution, regardless how simple it is.
On a side note, it seems like many trending topics on HN are heavily bashing new technologies (see k8s, rust, clouds, etc...) that are already widely adopted. Most communicate a resignation from learning something too complex. Docker is not complex, k8s is not complex, rust is not complex; the problems they are addressing are. Most teams developing things that run on the web will have to deal with these problems hopefully before they don't make it. The rest are wizards like the stackoverflow team.
I don't think it's about "bashing new technologies" so much as it is about offering another perspective. Perhaps one that's less sexy, or perhaps one that's simpler and might work just as well in some cases. "XYZ SUCKS" posts are few and far between.
I strongly suggest a container or VM for building because of hygiene. A Golang binary is not completely standalone (do a ldd if you don't believe me). Just copying might lead to bizarre runtime errors. And if course, if you are truly paranoid, there is the kernel ABI to consider.
But more likely than that you might end up with developer artifacts (environment variables, paths) inside your binary. So at least guard against this before shipping.
But more likely than that you might end up with developer artifacts (environment variables, paths) inside your binary. So at least guard against this before shipping.
Docker is not for production (most of the time), it's a dev tool.
Ignore docker compose (and swarm and whatever else), because you don't need it. The basic Dockerfile format is simple, it reads like a bash script without logic, basically it's a list of dependencies. Then you need to understand the difference between images and containers, and you're good to go.
If you're lucky enough to be developing on Linux, then docker is just a name for a bunch of stuff that's mostly already built into the OS.
Ignore docker compose (and swarm and whatever else), because you don't need it. The basic Dockerfile format is simple, it reads like a bash script without logic, basically it's a list of dependencies. Then you need to understand the difference between images and containers, and you're good to go.
If you're lucky enough to be developing on Linux, then docker is just a name for a bunch of stuff that's mostly already built into the OS.
I use docker compose only. It is much more convenient than vanilla docker.
> Docker is not for production (most of the time), it's a dev tool.
Sure.
>Ignore docker compose
Don't ignore Docker compose. It allows you to set up a test/dev environment with dependencies between containers, shared variables/volumes, etc.
>If you're lucky enough to be developing on Linux, then docker is just a name for a bunch of stuff that's mostly already built into the OS.
We could all just use cgroups to achieve containerization, but why? Docker is a great tool with a huge ecosystem of images at your fingertips, mature and powerful command-line interface, and lots of documentation/googlable discussions for even the most esoteric problems.
But yeah, I wouldn't use it in production.
Sure.
>Ignore docker compose
Don't ignore Docker compose. It allows you to set up a test/dev environment with dependencies between containers, shared variables/volumes, etc.
>If you're lucky enough to be developing on Linux, then docker is just a name for a bunch of stuff that's mostly already built into the OS.
We could all just use cgroups to achieve containerization, but why? Docker is a great tool with a huge ecosystem of images at your fingertips, mature and powerful command-line interface, and lots of documentation/googlable discussions for even the most esoteric problems.
But yeah, I wouldn't use it in production.
> dependencies between containers
This is the part I don't understand. I use Docker as a light weight alternative to say VMWare or Virtual Box, just Dockerfiles work well for this use case. The point is to not have dependencies
This is the part I don't understand. I use Docker as a light weight alternative to say VMWare or Virtual Box, just Dockerfiles work well for this use case. The point is to not have dependencies
Sorry, but the stuff you just said is bullshit. Any Docker user will hopefully see that.
For remotely any service requiring configuration (database, search index, language runtime, cache, etc.) across a team of 3-4+, it would behoove the team to maintain a common containerized setup in which to develop. Otherwise, you end up with package/dependency fragmentation, and the classic “works on my machine” problems.
There’s absolutely no shame in a minimal environment for production, but let’s not be so bold as to make developers deal with the mess that usually results in local dev environments.
There’s absolutely no shame in a minimal environment for production, but let’s not be so bold as to make developers deal with the mess that usually results in local dev environments.
> No, the reason we went with go was because golang’s packaging is so much better than node. Or java. Or C#. We get a single binary. Building is as easy as: `go build`
This is only true insofar as you have an extended group of collaborators committed to observing the fundamental tenets[1] that make this work. Not hard to run into (popular!) software written in Go that is as bereft of this simplicity as the other ecosystems the author compares to. (Interestingly, Docker itself is the product of minds that were exposed to Go, and yet these principles didn't "take" in that case...)
1. http://blogs.perl.org/users/leon_timmermans/2020/08/perl7-is...
This is only true insofar as you have an extended group of collaborators committed to observing the fundamental tenets[1] that make this work. Not hard to run into (popular!) software written in Go that is as bereft of this simplicity as the other ecosystems the author compares to. (Interestingly, Docker itself is the product of minds that were exposed to Go, and yet these principles didn't "take" in that case...)
1. http://blogs.perl.org/users/leon_timmermans/2020/08/perl7-is...
Docker is not the criteria for CI/CD?
My previous company had a mix of docker and non-docker environments. We still had a CI/CD pipeline. SCP'ing a single binary still counts as an automated deploy pipe, not sure what that has to do with anything.
Also: Go is only truly statically compiled if you turn CGO off. Otherwise it's relying on system libs, which means you might be surprised if you upgrade a server and forget the other. I've been there :)
My previous company had a mix of docker and non-docker environments. We still had a CI/CD pipeline. SCP'ing a single binary still counts as an automated deploy pipe, not sure what that has to do with anything.
Also: Go is only truly statically compiled if you turn CGO off. Otherwise it's relying on system libs, which means you might be surprised if you upgrade a server and forget the other. I've been there :)
Yeah, I had to statically link the clibs for an open-source project because it didn't work for some folks. This is something a lot of Go devs keep forgetting about.
For small scales and small teams that know what each other are doing, what the author mentions is fine.
But can you seriously imagine onboarding a new engineer and telling them to deploy to prod, they need to run an `scp` command onto a host?
Docker solves a lot of problems, among them include packaging and deploying a universal 'deployment unit' that works across many systems.
The author only requires one uncontrolled unknown to be introduced to break their entire unicorn setup.
But can you seriously imagine onboarding a new engineer and telling them to deploy to prod, they need to run an `scp` command onto a host?
Docker solves a lot of problems, among them include packaging and deploying a universal 'deployment unit' that works across many systems.
The author only requires one uncontrolled unknown to be introduced to break their entire unicorn setup.
Agreed.
I get a lot of "engineering comfort" knowing that the environment I use on my machine to build, test and run an executable is the same as the one used in prod.
I get a lot of "engineering comfort" knowing that the environment I use on my machine to build, test and run an executable is the same as the one used in prod.
I used to work on a team that was trying to adopt Docker. I was pretty much the sole developer of the frontend and one of the top active developers on the backend. It was really overkill to install an entire VM and spin up my computer fans to run a simple frontend webapp that was `npm install && npm run start`.
Why not run it directly on the host machine during development, and run the docket image on the servers? This is the approach that I usually follow and it has worked fine for me so far. I need to build the Docker image locally only if something breaks down in the build pipeline.
Doesn't that sort of kill one of the main selling points of using docker in the first place?
Consistent environment on local, test and prod servers?
Consistent environment on local, test and prod servers?
I think they meant using a remote development server instead of spinning up a VM locally.
There was a post a few days back about using a NUC as a powerful remote (over LAN) development server
There was a post a few days back about using a NUC as a powerful remote (over LAN) development server
Well, their selling point was that it was easier to develop on locally. We weren't even running Docker in production.
You start as simple as you can, you add tech to the stack as needed after calculating the cost, benefit and risk.
I don't know why these topics are so controversial. For the code almost everybody accepts that you should deliver value fast and add complexity only when needed, why would it be different for infrastructure?
I don't know why these topics are so controversial. For the code almost everybody accepts that you should deliver value fast and add complexity only when needed, why would it be different for infrastructure?
Containers, like a lot of tech tools, is situational. You should use a tool because it solves a problem you're having, not because it's popular. There's a lot of value in keeping your pipeline simple, but there's also plenty of wisdom in using the right tools for the job.
Golang apps are super easy to dockerize (if you ever decide you need to do that). I build and deploy go apps to ECS Fargate with a simple Makefile and vim. And using Alpine means the final product is only a few megs larger than my go binary.
I'm often criticized (by the young devs) for not having an automated pipeline with a lot of bells and whistles, but honestly, the Makefile is all I need.
I do agree with the general concept of over-engineering. One of my former CTO's called it technological self-pleasure and ultimately a distraction.
I'm often criticized (by the young devs) for not having an automated pipeline with a lot of bells and whistles, but honestly, the Makefile is all I need.
I do agree with the general concept of over-engineering. One of my former CTO's called it technological self-pleasure and ultimately a distraction.
It's never as simple as "slap an exe and launch it" : even super simple MVCs need some server state ( lucky you if you don't ! ) : this means db migrations, configs ... Not that docker solves this though. Also I disagree with the "can't do this with java" : any JVM based framework offers self sustained jars packaging ( might be slightly bloated, OK ).
If I need a database, I slap an exe with sqlite, and launch it :)
I know it won't scale to write-heavy Enterprise OLTP workflows, but there are plenty of cases where it's good enough up to millions of rows and gigabytes of data.
I know it won't scale to write-heavy Enterprise OLTP workflows, but there are plenty of cases where it's good enough up to millions of rows and gigabytes of data.
I was expecting the sqlite comment ;-) still : you need to handle migrations, you may very well have other dependencies on external APIs or services ...
I agree with the Docker sentiment, but it kinda sounds like the argument here is "I don't need lemonade, I have ketchup instead."
I can also confirm that I've run multiple personal projects using something very similar. Basically I just deploy the binary as a `systemd` service on a VPS somewhere (Amazon or DigitalOcean traditionally) and I don't have to think about anything else. If it goes down I can rely on `systemd` to bring it back up, and I usually pair it with some kind of logging system such as Papertrail or Datadog.
That being said, some use cases do need more than just a binary running on a VPS. Recently I've been moving more towards the AWS stack. My issue with the AWS stack for personal projects is that the bill can get unwieldy very quickly. On my last personal project I actually ended up going for a cheap random VPS provider I found on Google and spun up my entire application for about $80/mo. Still pretty expensive, but it worked for my use case
That being said, some use cases do need more than just a binary running on a VPS. Recently I've been moving more towards the AWS stack. My issue with the AWS stack for personal projects is that the bill can get unwieldy very quickly. On my last personal project I actually ended up going for a cheap random VPS provider I found on Google and spun up my entire application for about $80/mo. Still pretty expensive, but it worked for my use case
Yup, when you have one production server or maybe two for good measure, you don’t need a big layer of abstraction on top.
“We don’t need to do very much so we don’t do very much” is a good place to be, what is hard is deciding what and when for building out abstractions. Almost everywhere does it way too early or way too late.
“We don’t need to do very much so we don’t do very much” is a good place to be, what is hard is deciding what and when for building out abstractions. Almost everywhere does it way too early or way too late.
Good point about timing: it really is easy to spend your wheels and waste time early in the app lifecycle on things that don’t matter and never will. But it’s also true that if you stay focused on cranking out product and don’t worry about long-term architecture, you just might end up with a large, successful ball of mud that’s no fun to maintain or work on. Finding the inflection point between those modes isn’t easy.
What I have observed over and over is the conspicuous lack of the question “is this the right tool for the job right now?”
Docker is like MongoDB. It was cool for a while, until everybody realized you don't actually need it.
Well, I'm not a fan of docker, but I do like Mongo.
MongoDB marries well with Node, and things like json-patch.
Especially great when dealing with graph-like structures, though jsonb data types in recent rdbms' have gone a long way towards evening the playing field there.
I've also come to appreciate Mongo's pipeline approach to aggregation. I hated it with a passion at first, but when coupled with Compass, I've found it makes aggregation stages much easier to reason about than the equivalent SQL.
MongoDB marries well with Node, and things like json-patch.
Especially great when dealing with graph-like structures, though jsonb data types in recent rdbms' have gone a long way towards evening the playing field there.
I've also come to appreciate Mongo's pipeline approach to aggregation. I hated it with a passion at first, but when coupled with Compass, I've found it makes aggregation stages much easier to reason about than the equivalent SQL.
If you use 100% Go, Docker doesn’t give you much.
But if you use other languages where library dependencies is an art form, then Docker is a life saver.
But if you use other languages where library dependencies is an art form, then Docker is a life saver.
> Writing golang web services and static html embedded with with golang 1.16’s new //embed directive, we end up with a single deployable binary.
So you don't use Docker because you use an alternative technology to achieve exactly the same thing?
Docker is one way to encapsulate a service into a single deployable component with a standard interface. Building a single ELF via static linking is another way. But that's not possible if you use Python or Ruby on Rails or many other popular web technologies is it?
So you don't use Docker because you use an alternative technology to achieve exactly the same thing?
Docker is one way to encapsulate a service into a single deployable component with a standard interface. Building a single ELF via static linking is another way. But that's not possible if you use Python or Ruby on Rails or many other popular web technologies is it?
[deleted]
I don't know about how hermetically sealed go's packaging is, but the main advantage of docker is that when Ubuntu 26.04 and python4.3 comes out your docker image containing nodejs, python3, and tensorflow 2.3 code should theoretically still just work out of the box. Really good for "set and forget" deployments that might be forced onto new hardware or OS at some point but when you don't want to ever touch the code again except for bugfixes.
Go's packaging is about as hermetically sealed as it gets--deployment is a single executable binary. No external runtime, no support files.
What if you need to use e.g. MongoDB or Redis or Nginx or CUDA or CUDNN as part of your deployment? I imagine Go isn't going to package all of that stuff into a binary though.
This is why I like docker for the personal servers I run - compose file pointer to a pinned docker file on a nas and boom: pihole, dns,dhcp, Emby, ddns, hass, whatever - easy to backup, view/edit and stable. Used to run everything as a service and it was never this easy.
Not that this says anything about running your own code, but abstracting away everyone else’s multitude of deps and configs is pretty neat!
Not that this says anything about running your own code, but abstracting away everyone else’s multitude of deps and configs is pretty neat!
What amazes me is that in 2021, the big server/desktop OSes still don't use this kind of set-up as standard. Shouldn't the norm today be each service or application running in some controlled and sandboxed environment, at a specific version of the software, with access to system resources and networking explicitly specified, and with everything other than deliberately persistent resources disposable if you want to update to a new version or remove some software from your system entirely? It seems like this would solve all kinds of long-standing problems with security, privacy, system stability, etc.
Tools like Docker might still be useful for portability even in that environment, for example if you were developing on one OS and running production on another. However, it feels like packaging/installing/updating all applications and services could be similar to how we build and manage lightweight containers/jails/VMs as developers/sysadmins. Install a package, then provide access to networking and storage and other local or remote resources as appropriate for each case.
There could still be user-friendly interfaces and app stores and all that stuff on top for non-geeks to install and update things or see what is happening if they want to. The OS and application developers could provide sensible defaults that would probably work in almost all cases for "normal" users anyway, so most of the time I wouldn't expect an average user to really be aware of what was happening under the surface.
However, it seems crazy to me that here we are with all this modern technology and we can still have basic questions about what is actually running on our laptop or even a production server, and where it came from, and how up to date it is, and whether anything still needs it to be there anyway, and whether it's secretly encrypting our files for ransom or spying on us using our webcam and uploading the footage or just nuking yesterday's work document because our favourite game's save code had a bug and overwrote a file in the wrong place.
Tools like Docker might still be useful for portability even in that environment, for example if you were developing on one OS and running production on another. However, it feels like packaging/installing/updating all applications and services could be similar to how we build and manage lightweight containers/jails/VMs as developers/sysadmins. Install a package, then provide access to networking and storage and other local or remote resources as appropriate for each case.
There could still be user-friendly interfaces and app stores and all that stuff on top for non-geeks to install and update things or see what is happening if they want to. The OS and application developers could provide sensible defaults that would probably work in almost all cases for "normal" users anyway, so most of the time I wouldn't expect an average user to really be aware of what was happening under the surface.
However, it seems crazy to me that here we are with all this modern technology and we can still have basic questions about what is actually running on our laptop or even a production server, and where it came from, and how up to date it is, and whether anything still needs it to be there anyway, and whether it's secretly encrypting our files for ransom or spying on us using our webcam and uploading the footage or just nuking yesterday's work document because our favourite game's save code had a bug and overwrote a file in the wrong place.
It also really surprises me that in 2021 we have L4 self-driving cars but not L4 self-installing software that can figure out how to run itself. You'd think the latter to be much easier but even NVIDIA's own stuff horribly fails.
Try linking against glibc and then run this Go binary in musl env ;-)
Containers allow you to choose your resolver, libc, etc.
Python 4.3 should definitely run in a container.
Python 4.3 should definitely run in a container.
you can do the same with dotnet.
not using docker has nothing to do with a simple ci/CD solution. having pull requests run tests and then creating releases especially when you have a single binary should be dead easy and a very basic part of your Development life cycle.
I don't use docker in production, but use it heavily on developers machines or to setup infra during QA.
The article doesn't really say anything except "look at me not using docker you haters".
not using docker has nothing to do with a simple ci/CD solution. having pull requests run tests and then creating releases especially when you have a single binary should be dead easy and a very basic part of your Development life cycle.
I don't use docker in production, but use it heavily on developers machines or to setup infra during QA.
The article doesn't really say anything except "look at me not using docker you haters".
A lot of the confusion comes from the hazy definition of what "Docker" is. Not helped by the fact that Docker-the-company uses it for some pretty wildly varying things. Eg this post talks about "deploying Docker"... probably they're not talking about the Linux dev tool /usr/bin/docker anymore (or is it called Moby now?), since you usually don't use that to deploy in production.
A big problem these days is that you have to spend a week or even more getting your pipeline working.
The problem domain should not be that difficult.
There should be a lot more automation and standards.
It should be worth missing some customization or perhaps making the flow with the animation instead of everything being bespoke.
You can probably write a faster or better suited for a project over using whatever library your programming language needs, most accept the trade offs with using the most generic ones.
There is also ah explosion in the number of products used for each project. From projects I have been on lately that adds a hell of a lot of confusion and complexity for deployment and makes the solution overall Esp since most of the products have words and issues you need to be aware of.
Last project they spent 3 weeks getting the pipeline going. For the last week and a half they brought in an expert in AWS to help them at a HUGE cost. Even he had tea trouble getting it working. We “had” to do this because “devOps” and save money buy not paying for “old fashioned not needed infrastructurepeole”
The system were creating was not very difficult not would it ever need to scale much at all. . Cargo culting can be very harmfu.
The problem domain should not be that difficult.
There should be a lot more automation and standards.
It should be worth missing some customization or perhaps making the flow with the animation instead of everything being bespoke.
You can probably write a faster or better suited for a project over using whatever library your programming language needs, most accept the trade offs with using the most generic ones.
There is also ah explosion in the number of products used for each project. From projects I have been on lately that adds a hell of a lot of confusion and complexity for deployment and makes the solution overall Esp since most of the products have words and issues you need to be aware of.
Last project they spent 3 weeks getting the pipeline going. For the last week and a half they brought in an expert in AWS to help them at a HUGE cost. Even he had tea trouble getting it working. We “had” to do this because “devOps” and save money buy not paying for “old fashioned not needed infrastructurepeole”
The system were creating was not very difficult not would it ever need to scale much at all. . Cargo culting can be very harmfu.
I too love how simple the infrastructure can get using Go’s tools. At the same time, I prefer Node.js for the libraries & bigger community. So I developed a tool called caxa which brings the best of both worlds, packaging any Node.js into a single binary. Check it out at https://github.com/leafac/caxa
Debugging Docker? You are like a little baby.
Once you grow up, you can start debugging kubernetes, kustomize and helm. YAMLs on top of YAMLs rewriting other YAMLs.
Once you grow up, you can start debugging kubernetes, kustomize and helm. YAMLs on top of YAMLs rewriting other YAMLs.
I agree with the author post. I'm an early Docker user in 2013 and use it heavily everywhere I can. However, I see the author point. Pretty much what we can achieve with docker can be achieve with systemd.
Something is also really hard with docker is mail server. I run an email forwarding service [0] and use Postfix. Postfix requires opendkim, rspamd, opendmarc etc...now once I puts them into a container suddenly I have to have a supervisor to manage them, if I run them separately then I have a put myself to hell to manage the graph mess.
I think any services that has a few own sub dependencies shoulnd't use Docker at all.
Docker works great for stuff where you have a run time(think Ruby, python, node etc). It makes upgrading and vendor that run time easily. You can mix and matched version. Say Ruby, it's doable with rbenv/rvm on the host but quite complex to mess around with path.
---
[0] https://hanami.run
Something is also really hard with docker is mail server. I run an email forwarding service [0] and use Postfix. Postfix requires opendkim, rspamd, opendmarc etc...now once I puts them into a container suddenly I have to have a supervisor to manage them, if I run them separately then I have a put myself to hell to manage the graph mess.
I think any services that has a few own sub dependencies shoulnd't use Docker at all.
Docker works great for stuff where you have a run time(think Ruby, python, node etc). It makes upgrading and vendor that run time easily. You can mix and matched version. Say Ruby, it's doable with rbenv/rvm on the host but quite complex to mess around with path.
---
[0] https://hanami.run
Yes, simplicity is vastly underrated. Our machines are amazingly capable and can take amazing loads if we let them.
Yeah, back in the early 2000s I was asked by the BBC to help with one of their backend systems that was spread over a dozen machines running a hundred processes and connected to their enterprise Oracle.
It was incredibly slow and fell over daily. Also pretty much unmaintainable.
Replaced it with a single jar. Deployment process: copy jar to production machine, start it. Was 100-1000 times faster and also 100-1000 times more reliable. We initially did have a failure because we were all Java noobs and didn't know we had to set a JVM parameter to get it to use all the gigabyte of RAM on its dedicated machine. After that: rock solid.
¯\_(ツ)_/¯
Yeah, back in the early 2000s I was asked by the BBC to help with one of their backend systems that was spread over a dozen machines running a hundred processes and connected to their enterprise Oracle.
It was incredibly slow and fell over daily. Also pretty much unmaintainable.
Replaced it with a single jar. Deployment process: copy jar to production machine, start it. Was 100-1000 times faster and also 100-1000 times more reliable. We initially did have a failure because we were all Java noobs and didn't know we had to set a JVM parameter to get it to use all the gigabyte of RAM on its dedicated machine. After that: rock solid.
¯\_(ツ)_/¯
I'm all for working to avoid docker in your local development environment if you can swing it. Docker is a beast and if it's way faster to `go run` then do it.
But docker is the deployment artifact of our age. Simple go binary? great! but don't deploy that with SCP, put it in a bare bones docker image because that is the language of deployment today.
I've been using the Serverless framework to deploy to lambda recently, and if I could change one thing it would be to deploy using a docker image instead of the old lambda zip file. The docker image would be versional in a way that is difficult with the serverless framework presently.
But docker is the deployment artifact of our age. Simple go binary? great! but don't deploy that with SCP, put it in a bare bones docker image because that is the language of deployment today.
I've been using the Serverless framework to deploy to lambda recently, and if I could change one thing it would be to deploy using a docker image instead of the old lambda zip file. The docker image would be versional in a way that is difficult with the serverless framework presently.
in the long run this is gonna hurt you. Theres a middle ground between
> It sure would be nice if we could spend a couple weeks building the perfect CI/CD pipeline, an elegant deployment process, along with pretty dashboards.
And having at least a simple container to lock in dependencies and operating system. You could set one up in an hour if your application is truly so simple. Then from this container you could iterate in the future. How can you truly trust your own development environment when you aren't using docker?
It's odd to say you don't need docker when you use systemd to solve a lot of the same problems. It's not that you don't need what docker provides, it's that you're getting it elsewhere. It's not like systemd is far less complex, either.
Docker containers are really just bundled-up processes. Once you move beyond things that come in statically linked binaries, it's nice to have a common packaging format, especially one that has such an expansive ecosystem and buy-in from every major cloud provider.
Docker containers are really just bundled-up processes. Once you move beyond things that come in statically linked binaries, it's nice to have a common packaging format, especially one that has such an expansive ecosystem and buy-in from every major cloud provider.
I must admit I Dockerize almost everything that has more than a couple build and install steps, including the programming language itself. It’s just much easier to share with teams.
Not really related to the debate but I highly recommend Watchtower [1] when using Docker in production.
This library, that you can plug as a service in any docker-compose, will check your Docker registry to see if the images used by your containers has been updated, pull the new images and restart your containers if needed.
That makes deployment, for some cases, really nice.
[1] https://github.com/containrrr/watchtower
This library, that you can plug as a service in any docker-compose, will check your Docker registry to see if the images used by your containers has been updated, pull the new images and restart your containers if needed.
That makes deployment, for some cases, really nice.
[1] https://github.com/containrrr/watchtower
C# (dotnet core) allows publishing self-contained executables with single line command, there are no much difference there
Except generics already present in the language ;)
I worked a on a couple teams where there was a constant crush to reduce complexity, remove code, streamline, remove corner cases. What is even better, is when engineers can be product designers and simplify features or cut them entirely. My best example of this when the spec called for a continuous value for a predicate, I arbitrarily changed it to small, medium and large. But it brought in the schedule and the schedule by 10 days.
Shared simplicity is a superpower.
Shared simplicity is a superpower.
We don't need it 'now' - maybe the right way
-- If choice is due to complexity of dockers loc as was mentioned, then that should be compared against stability; as abstractions go, it is not leaky and pretty stable -- Now-a-days it hardly takes a few minutes to use docker or podman; and few more to configure a free CI system; and you have a good base to start and grow and migrate to k8s or such when your services grow
-- If choice is due to complexity of dockers loc as was mentioned, then that should be compared against stability; as abstractions go, it is not leaky and pretty stable -- Now-a-days it hardly takes a few minutes to use docker or podman; and few more to configure a free CI system; and you have a good base to start and grow and migrate to k8s or such when your services grow
I mean, at this scale, why aren't you at least running in Heroku so you can have some niceties and not have to scramble when your VPS craps the bed?
There is a middle ground. At sourcehut we don't use Docker, but we do use CI/CD and automate deployments. We have something like 12 dedicated hosts colocated in a local datacenter and we use virt-manager to host virtual machines - basically self-hosted VPS lite. It's a lot simpler and more robust, and easier to keep everything up to date. Fewer moving parts == more reliable software.
Have you considered using lxc or lxd to run containers rather than VMs? That way you could have a separate rootfs, network interface, etc. for each service, without the virtualization tax.
It took a while to know I needed it but eventually I found learning Docker and using docker-compose is definitely worth the investment. Just being able to run a script and startup revision controlled services like Redis, imgproxy, elasticsearch, kibana, fluentd, and ensure its the same on multiple computers, it's just so much easier to manage.
Docker/Containerization is extremely useful for development, CI workflows and integration testing. So everyone can spin up the complete system within a few minutes in a defined state.
To use it in production is quite a different story. KISS may work there way better, than k8s in combination with 10 other "innovative" technologies...
To use it in production is quite a different story. KISS may work there way better, than k8s in combination with 10 other "innovative" technologies...
Other then I use C++ instead of Go they have just described my exact approach to software development. I do not develop for FAANG and for what I do for my clients it is hard to find the task that modern monsters of computer can not solve by way of vertical scaling.
I did some map reduce type projects but those were very special cases.
I did some map reduce type projects but those were very special cases.
A reasonable use of docker I've seen in the wild is for pluggable machine learning algorithms for aws sagemaker
Otherwise it's just somewhere between a chroot jail and a full virtual machine
From what I understand, the containers share the host's kernel, and can share each other's userspace if same version, yes?
Otherwise it's just somewhere between a chroot jail and a full virtual machine
From what I understand, the containers share the host's kernel, and can share each other's userspace if same version, yes?
Coincidentally I'm currently rewriting a flask application that runs on docker in Golang. The application doesn't use any kind of file or persistent data. During the process I've been thinking "Do I really need to run this on docker?".
What is mentioned in the article is exactly the same reason we are not using it either. Even though we use Java and F# it is just too easy to package up these to a single binary and create an Ansible script that deploy these.
Author, it's not the only job you have in your life. Learning new technologies will improve your resume (and will let you to look at the problem from different angles, will help you to understand other engineers).
Why edit the submission title, mod(s)? It matched the actual article before.
Docker is fantastic, definitely improved my development process for Python.
The author is comparing a programming language to a container runtime. I don't think he/she understands the big picture of what problems Docker actually solves.
It's all fun and games until your next whale of a customer asks for your SOC report. Then the auditors are asking you to prove you've audited your code and dependencies. They ask "who has access to customer data? Who can deploy code? What about your perimeter security and intrusion detection? Are you running virus scanners on all of your servers?"
Then you realize it's easier to have a complex deploy pipeline that does all this stuff that isn't just writing your app for you. And containers are a decent enough way to abstract your app away from all the cruft.
I guess the answer is don't write enterprise software?
Then you realize it's easier to have a complex deploy pipeline that does all this stuff that isn't just writing your app for you. And containers are a decent enough way to abstract your app away from all the cruft.
I guess the answer is don't write enterprise software?
I do basically the same with python. The host does need to have the interpreter installed, then I can just rsync my local venv, not even a build step is neccessary.
I think this is particularly dangerous with python. Some of the things it installs will compile c dependencies for the architecture you are on, I thought?
Not only architecture. You can have either the compiled dependencies linked with external libraries or dependencies loading libraries via ffi. Deploying python code like that is at best accidentally correct. (Unless you really check for those issues on every dependency change)
Yeah definitely the case with numpy and libstdc++ and others. I cook up a Nix-shell when I need those. Already having those dependencies is pretty common in FHS linux systems in my experience.
It's usually pre-compiled but yes. Go binaries are also architecture-specific, so it's equivalent there. I'm fairly sure my whole venv is pure python though so it should be even more portable if you factor out the interpreter.
Cool, glad I'm not completely off my ticket.
My exposure, I thought, was in pandas and friends compiling a blas library. I'd expect that to be fairly dependent on code generated. And, I don't think there is a blas for go, yet. Is there? That said, fair point that it will also be dependent on compiling for the lowest target.
My exposure, I thought, was in pandas and friends compiling a blas library. I'd expect that to be fairly dependent on code generated. And, I don't think there is a blas for go, yet. Is there? That said, fair point that it will also be dependent on compiling for the lowest target.
It's not that bad, as long as you are aware of https://www.python.org/dev/peps/pep-0599/ and related.
A locally hosted instance of devpi where pre-built python packages with the C dependencies compiled can get around that problem. Alternatively, the python application can be packaged with dependencies specified and can be installed using the distro package manager (with dependencies pulled from the same package repositories).
>We specifically made choices to avoid needing a database (yet). State is the root of all evil. ;)
No DB needed? I am curious how this is achieved.
No DB needed? I am curious how this is achieved.
The comic looked like abstrusegoose.com. Sometimes his puns are subtle.
This one is not.
8th frame should have been like 4th frame.
I file this under "cool story bro"
Nothing interesting or thought provoking was said in this article.
Sometimes I miss the simplicity of working at a small startup.
The irony of the link being down right now is great.
pretty amazing how this holds. I wonder if there are availability data and how do they deal with regular upgrades
[deleted]
But docker is webscale.
Yikes, static builds to the rescue.
But: well, administrators like to administrate.
But: well, administrators like to administrate.
A lot of programmers talk about K.I.S.S (Keep It Simple Stupid) when it comes to writing their software but don't extend that to their build tools, or how they deploy it. Just like the "you are not google talk"^1 people tend to overengineer solutions for a problem they could have in the future. I'm certainly guilty of this. At work I can't just write some python code in an editor, I have to write and pass around Jupyter Notebooks. Or worse I have to launch a Jupyter Notebook inside a Docker container so that it loads the correct dependencies. I understand this is done to make things more reproducible but we've still had issues getting clients set up in our workflow even using this method.
I will be the first to admit I am a noob when it comes to systems programming, but I found it so frustrating when I attempted to learn C and C++ having to download and use Make, Cmake, Ninja etc, vcpkg, just to interface with other peoples code. And it was never just as simple as running the install script. A great example is Raylib on the PI4. The majority of Pis are running Raspbian/Raspberry PI OS and the only thing that changes between PI 4s is the amount of memory. So this should be the best case scenario for creating a install script. The raylib documentation says it should only require a couple Make commands. I ran them and it still failed.
I appreciate how D and Rust both have Cargo and Dub. While they have their warts, it makes it sooo much easier to link and install other peoples code, and import/use it in your code. When the languages and problems are hard enough, I don't want to be futzing around with linking and dependency management. With how much people complain about software licensing, because they don't want to have to thing about the code they are using in their project, I think about the install process.
And in regards to things being slower and breaking now a days compared to back in the day... I can't comment on that. I'm too young to remember what software was like back in the 80s and 90s, I just hear what older people tell me. I know Casey Muratori has an example of the watch window in Visual Studio being worse today then it was in the Windows XP^2 days. There are probably other examples. What I do know is this. Programs I open on my desktop fail to open, or crash unexpectedly all the time. A great example of this is Webex Teams, which I have to use everyday at work.
I don't think people question that software in some cases is doing much more complicated stuff then it was doing 30 years ago, but I think people also have an expectation that it would be more stable than it was 30 years ago. I mean... People used to write programs on computers without memory protection! When before I can use a piece of software I have to sign in to my account, which has to authenticate in the cloud, that software is objectively slower than the version that I could use which did not require a login, but for me to purchase the actual software. It doesn't matter how much better, or more complex it is now, that process only exists so that the Company can fight piracy, and exert control over the product. It does not serve Me.
1.https://blog.bradfieldcs.com/you-are-not-google-84912cf44afb
2. https://youtu.be/GC-0tCy4P1U?t=2170
I will be the first to admit I am a noob when it comes to systems programming, but I found it so frustrating when I attempted to learn C and C++ having to download and use Make, Cmake, Ninja etc, vcpkg, just to interface with other peoples code. And it was never just as simple as running the install script. A great example is Raylib on the PI4. The majority of Pis are running Raspbian/Raspberry PI OS and the only thing that changes between PI 4s is the amount of memory. So this should be the best case scenario for creating a install script. The raylib documentation says it should only require a couple Make commands. I ran them and it still failed.
I appreciate how D and Rust both have Cargo and Dub. While they have their warts, it makes it sooo much easier to link and install other peoples code, and import/use it in your code. When the languages and problems are hard enough, I don't want to be futzing around with linking and dependency management. With how much people complain about software licensing, because they don't want to have to thing about the code they are using in their project, I think about the install process.
And in regards to things being slower and breaking now a days compared to back in the day... I can't comment on that. I'm too young to remember what software was like back in the 80s and 90s, I just hear what older people tell me. I know Casey Muratori has an example of the watch window in Visual Studio being worse today then it was in the Windows XP^2 days. There are probably other examples. What I do know is this. Programs I open on my desktop fail to open, or crash unexpectedly all the time. A great example of this is Webex Teams, which I have to use everyday at work.
I don't think people question that software in some cases is doing much more complicated stuff then it was doing 30 years ago, but I think people also have an expectation that it would be more stable than it was 30 years ago. I mean... People used to write programs on computers without memory protection! When before I can use a piece of software I have to sign in to my account, which has to authenticate in the cloud, that software is objectively slower than the version that I could use which did not require a login, but for me to purchase the actual software. It doesn't matter how much better, or more complex it is now, that process only exists so that the Company can fight piracy, and exert control over the product. It does not serve Me.
1.https://blog.bradfieldcs.com/you-are-not-google-84912cf44afb
2. https://youtu.be/GC-0tCy4P1U?t=2170
tl;dr
Static go/rust executables make deployment easy
Static go/rust executables make deployment easy
Yep. I feel like there’s a weird war between static and dynamic linking. Distro maintainers love dynamic linking because they can upgrade one dynamic library to fix a security issue, rather than pushing security patches to everything it depends on. Software houses love static linking because they (we) want to know exactly what version of every dependency our software uses when it runs. Docker forces an entire userland to be statically distributed - but everything it does should be possible using static linking + some tooling to easily ship a binary.
Playing this turf war out through tooling seems silly. Linux should support both without needing to rely on Docker and friends.
Playing this turf war out through tooling seems silly. Linux should support both without needing to rely on Docker and friends.
Distro builders and users also love dynamic linking because it saves so many resources. One of the things that really bother me with containerised desktop apps like snap.
Yep, that is really only when there is no maintainer for the package and you still need the software. Hope it does not become a reason for maintainers to throw in the towel on packages.
Well, except those groups don't particularly need dynamic or static linking, respectively, either.
Distro maintainers definitely loved dynamic linking 25 years ago, when they relied on a machine under someone's desk at e.g. Pixar for infrastructure and on the kindness of random universities for file hosting. It isn't really as true of distro maintainers today. You can update a library to fix a security issue and let a build farm handle the upgrade for pretty cheap (either using $your_favorite_public_cloud, or using a handful of relatively cheap physical servers if you prefer that). You can then upload the results to a CDN for free / cheap, who can get the modified binaries to your users quickly, without you being, as they used to say, "slashdotted." (There's also better technical tooling for things like binary diffs these days.) The cost of storage is also much cheaper, so you can save the intermediate build products if you want - generally you just need to rebuild one .o file and one .a file in a library and you can continue all the downstream builds from the linking step onwards, without recompiling anything else.
Software houses, meanwhile, are quite capable of shipping trees of compiled binaries. You can rebuild a bunch of shared libraries if you want, and ship them all together (Facebook, for instance, has a wonderfully terrifying system where they deploy a squashfs image and they have a FUSE filesystem to mount it: https://engineering.fb.com/2018/07/13/data-infrastructure/xa...). You don't actually need to statically link things to avoid OS dependencies; you've got $ORIGIN-relative rpaths and other means of finding dependencies relative from a binary. Or you can use a system like Nix where each version of a dependency has its own unique path, so you can co-install all sorts of things "systemwide" without them actually conflicting with each other, and the final binary was built against the paths to the specific exact dependencies you told it to use.
To be clear, I agree that this is a silly tooling turf war - I just want to encourage people to explore more tooling options and not feel like they're constrained by which "side" they fall on. There's a lot of options out there!
Distro maintainers definitely loved dynamic linking 25 years ago, when they relied on a machine under someone's desk at e.g. Pixar for infrastructure and on the kindness of random universities for file hosting. It isn't really as true of distro maintainers today. You can update a library to fix a security issue and let a build farm handle the upgrade for pretty cheap (either using $your_favorite_public_cloud, or using a handful of relatively cheap physical servers if you prefer that). You can then upload the results to a CDN for free / cheap, who can get the modified binaries to your users quickly, without you being, as they used to say, "slashdotted." (There's also better technical tooling for things like binary diffs these days.) The cost of storage is also much cheaper, so you can save the intermediate build products if you want - generally you just need to rebuild one .o file and one .a file in a library and you can continue all the downstream builds from the linking step onwards, without recompiling anything else.
Software houses, meanwhile, are quite capable of shipping trees of compiled binaries. You can rebuild a bunch of shared libraries if you want, and ship them all together (Facebook, for instance, has a wonderfully terrifying system where they deploy a squashfs image and they have a FUSE filesystem to mount it: https://engineering.fb.com/2018/07/13/data-infrastructure/xa...). You don't actually need to statically link things to avoid OS dependencies; you've got $ORIGIN-relative rpaths and other means of finding dependencies relative from a binary. Or you can use a system like Nix where each version of a dependency has its own unique path, so you can co-install all sorts of things "systemwide" without them actually conflicting with each other, and the final binary was built against the paths to the specific exact dependencies you told it to use.
To be clear, I agree that this is a silly tooling turf war - I just want to encourage people to explore more tooling options and not feel like they're constrained by which "side" they fall on. There's a lot of options out there!
the same can be said for the JVM; all I have is a jar for my product, and setting up a new machine is as easy as spinning up a VM, installing the JVM, then running a jar. It's great!
[deleted]
I just am starting to move off node/docker into deno/compiled binary and its great just having a binary run with no other dependencies. compile and start as service. rsync files up and you're good to go. I also am using sqlite and nedb instead of mongo or maria.
I've been preaching for years and every time I do I receive a lot of hate, also misinformation like "with Docker you can scale out effortlessly" which isn't a Docker feat at all.
For Go I copywrite a systemd service, for js/html I write a client side bash script and use rsync and run commands via ssh.
I never needed Docker. All it does is complicate your life.
Oh and for the record, I don't use overpriced Cloud either.
For Go I copywrite a systemd service, for js/html I write a client side bash script and use rsync and run commands via ssh.
I never needed Docker. All it does is complicate your life.
Oh and for the record, I don't use overpriced Cloud either.
Who actually runs Docker directly in production? Doesn't everyone just use managed solutions like ECS?
We run it (Docker Swarm) extensively inhouse at $dayjob, it runs loads of core services that are critical to operations. It's never given us any grief and is very easy to manage and maintain.
If the complexity ever gets to the point that k8s is justified we'll move there, but you can get a lot of work done with the much simpler Swarm.
If the complexity ever gets to the point that k8s is justified we'll move there, but you can get a lot of work done with the much simpler Swarm.
Docker Swarm usage here in production. Mainly a regulatory and compliance constraint, but it is relatively simple compared to other container orchestration platforms.
It runs very well on resumes. Seriously, I interviewed a guy last night and he wants 400k a year. I asked why, he said it because he is a container expert.
Apparently I am underpaid and this must be in the valley, right? That should be standard technology for an SRE by now. I have been using Docker since 2013, and I rarely encounter developers who don't have at least basic knowledge.
Once you dive really deep into Kubernetes with service meshes etc., that's where it gets interesting.
He is not from Valley. He is from Midwest.
"in the end can you really say it’s simpler than what we’ve got for free built in to golang?"
In this cherry picked example with a golang binary, sure. At my job, docker has saved us probably hundreds of man-hours. We don't have to worry about complications like systemd, because I can tell a container exactly how to run. Etcetera.
You can copy the golang binary into an alpine container and instead of having to maintain a server, you now have ephemeral containers that can be destroyed, spun up, scaled...
There is a difference of keeping things minimal and refusing to use industry tools. There is nothing scary about docker or even kubernetes. But they do take time to learn and understand, and yes, the benefits of containerization outweigh using bespoke server setups.
edit: s/systemd init scripts/systemd. We use s6-overlay instead of systemd to manage processes in containers.
In this cherry picked example with a golang binary, sure. At my job, docker has saved us probably hundreds of man-hours. We don't have to worry about complications like systemd, because I can tell a container exactly how to run. Etcetera.
You can copy the golang binary into an alpine container and instead of having to maintain a server, you now have ephemeral containers that can be destroyed, spun up, scaled...
There is a difference of keeping things minimal and refusing to use industry tools. There is nothing scary about docker or even kubernetes. But they do take time to learn and understand, and yes, the benefits of containerization outweigh using bespoke server setups.
edit: s/systemd init scripts/systemd. We use s6-overlay instead of systemd to manage processes in containers.
> We don't have to worry about complications like systemd init scripts
Setting up a systemd unit file (it's not even a script) doesn't even come close to the complexity of wrapping, maintaining, and potentially orchestrating software within containers.
> refusing to use industry tools
like systemd?
Setting up a systemd unit file (it's not even a script) doesn't even come close to the complexity of wrapping, maintaining, and potentially orchestrating software within containers.
> refusing to use industry tools
like systemd?
"Setting up a systemd unit file (it's not even a script) doesn't even come close to the complexity of wrapping, maintaining, and potentially orchestrating software within containers."
I find that the complexity of managing containers is overstated. It does have costs, but the benefits of docker and containerization in general outweigh those costs. Kubernetes is significantly more complicated and is not always a good choice depending on the use case.
"like systemd?"
I find that systemd tries to do far too much for our needs, namely for just managing services in a container. Our images have their services manged by s6-overlay [0], but there are multiple init systems one can use. I do not see the benefit of systemd over i.e. s6 or runit, but that is tangential to this discussion.
[0]: https://github.com/just-containers/s6-overlay
I find that the complexity of managing containers is overstated. It does have costs, but the benefits of docker and containerization in general outweigh those costs. Kubernetes is significantly more complicated and is not always a good choice depending on the use case.
"like systemd?"
I find that systemd tries to do far too much for our needs, namely for just managing services in a container. Our images have their services manged by s6-overlay [0], but there are multiple init systems one can use. I do not see the benefit of systemd over i.e. s6 or runit, but that is tangential to this discussion.
[0]: https://github.com/just-containers/s6-overlay
>You can copy the golang binary into an alpine container and instead of having to maintain a server
Why would you do this when a go binary doesn't need anything but the kernel? Why start with a full OS, even Alpine?
Why would you do this when a go binary doesn't need anything but the kernel? Why start with a full OS, even Alpine?
Alpine is a common example, but you could use scratch and only copy in exactly what you need. Both of these are extremely lightweight.
Having a full OS is helpful for troubleshooting issues, for starters. If one of the nodes is having an issue, it's helpful to be able to log in and examine the state of things.
One pain point of docker/k8s is figuring out networking, and having a container with dns-utils installed is quite handy.
Having a full OS is helpful for troubleshooting issues, for starters. If one of the nodes is having an issue, it's helpful to be able to log in and examine the state of things.
One pain point of docker/k8s is figuring out networking, and having a container with dns-utils installed is quite handy.
> We get a single binary.
Docker gives you a single container that can do 50 times more things. It's version controlled, it's IaC, immutable, automated, adds resource limits, networking flexibility, lets you run your build system and your production system on anything, etc. All of that and more is baked in. Literally nothing is baked into your solution other than your code.
> Note, we did get a little more complicated and create a SystemD script to launch it at startup.
Anyone who thinks using SystemD is less complicated than using Docker isn't familiar with either.
> Total time to create our build and deploy system was so small we don’t even know how to measure it.
Why would you brag about this? It's like bragging that you don't use Git because it's too complicated, and you just use cp --backup=t main.go archive/ a whole lot. Simple!
> Now, think about how much time you’ve spent learning docker. Deploying docker. Troubleshooting docker. Even if you love it, it changed your life, it’s better than sliced bread and it solves world hunger… in the end can you really say it’s simpler than what we’ve got for free built in to golang?
You've got nothing built into golang. Nothing.
I can build a single binary in a half dozen languages. That doesn't give me implicit network abstraction. It doesn't give me version controlled delta diffs of build images. It doesn't remove the need for Configuration Management. It doesn't give me the ability to expose different filesystems to the app when I run it, or secrets, or users, ora specific environment with specific other applications. It doesn't give me... Jesus, like a million other features. Go is just a static app.
I'm sorry, but as clickbait goes, this one was rather embarrassing. If you aren't comfortable running production services with Docker, that's fine! But this reads like your solution is almost better than using Docker. If you all knew how to use Docker you'd have a lot more useful functionality for free.
4 years ago, Docker was a shit show; it's pretty stable now. I really recommend spending the time to get the team comfortable with it.
Docker gives you a single container that can do 50 times more things. It's version controlled, it's IaC, immutable, automated, adds resource limits, networking flexibility, lets you run your build system and your production system on anything, etc. All of that and more is baked in. Literally nothing is baked into your solution other than your code.
> Note, we did get a little more complicated and create a SystemD script to launch it at startup.
Anyone who thinks using SystemD is less complicated than using Docker isn't familiar with either.
> Total time to create our build and deploy system was so small we don’t even know how to measure it.
Why would you brag about this? It's like bragging that you don't use Git because it's too complicated, and you just use cp --backup=t main.go archive/ a whole lot. Simple!
> Now, think about how much time you’ve spent learning docker. Deploying docker. Troubleshooting docker. Even if you love it, it changed your life, it’s better than sliced bread and it solves world hunger… in the end can you really say it’s simpler than what we’ve got for free built in to golang?
You've got nothing built into golang. Nothing.
I can build a single binary in a half dozen languages. That doesn't give me implicit network abstraction. It doesn't give me version controlled delta diffs of build images. It doesn't remove the need for Configuration Management. It doesn't give me the ability to expose different filesystems to the app when I run it, or secrets, or users, ora specific environment with specific other applications. It doesn't give me... Jesus, like a million other features. Go is just a static app.
I'm sorry, but as clickbait goes, this one was rather embarrassing. If you aren't comfortable running production services with Docker, that's fine! But this reads like your solution is almost better than using Docker. If you all knew how to use Docker you'd have a lot more useful functionality for free.
4 years ago, Docker was a shit show; it's pretty stable now. I really recommend spending the time to get the team comfortable with it.
SystemD is a lot less complicated than Docker is.
We run a service whose primary input is Dockerfiles; I'm not a Docker-hater. But doing the things we're talking about here in a systemd unit just isn't hard, and Docker, as you say, does a lot more than than systemd does.
Your comment would be better without the "rather embarrassing" swipe, for what it's worth.
We run a service whose primary input is Dockerfiles; I'm not a Docker-hater. But doing the things we're talking about here in a systemd unit just isn't hard, and Docker, as you say, does a lot more than than systemd does.
Your comment would be better without the "rather embarrassing" swipe, for what it's worth.
There's less code for SystemD, but it's more complicated to actually use, and less useful. Docker is user friendly, and the features scale with workflow changes where systemd doesn't.
And yeah, my comment was mean, I regret it.
And yeah, my comment was mean, I regret it.
Edit: Since people are confused, here is how StackOverflow handles of all of its web operations. If SO can run with this, so can your 0.33 req/minute app which is mostly doomed for failure. I am only half joking.
StackOverflow architecture, current load (it will surprise you): https://stackexchange.com/performance
Everytime you go to SO, it hits one of these 9 web servers and all data on SO sits on those 2 massive SQL servers. That's pretty amazing.
I want to be clear though, Horizontal scaling has a place in companies that has a team of corporate lawyers. Big. And in many many other scenarios for ETL and backend microservices.