I was able to reproduce this issue on kernel 6.12.57+deb13-amd64 running Debian 13 (Trixie), but unable to reproduce it on kernel 6.1.0-42-amd64 running Debian 12 (Bookworm).
For anyone not on the security stream of Debian packages for Bookworm, kernel version 6.1.0-42-amd64 is actually immune to copy.fail. Surprising that it looks to be immune to dirtyfrag. If you haven't already patched on the security stream, you can choose any kernel version that kept commit 2b8bbc64b5c2. I am thinking that the same commit might accidentally be keeping certain Debian 12 kernel versions safe from dirtyfrag as well.
> They don’t leave because, at the end of the day, they can’t allow themselves to risk their acquired prestige, which is another way of saying: they can’t risk losing external validation, the driving force of the prestige industrial complex.
Or they need to maintain their income to continue their lifestyle.
I am not particularly passionate about my job as a sysadmin. I don't hate it, but it would be nice to just spend all day skateboarding or playing Starcraft.
But, I have aging parents to take care of. Housing prices have skyrocketed around me. This is what I need to do to maintain a certain level of security, and indeed luxury, for myself and my close family.
And this isn't weird.
Go back 3-5 generations to when half of the population was employed in agriculture. Was everyone passionate about cows? Potatoes?
The reality is, I have a better shot at following my passion by trying to get into FAANG and working there for 10-20 years so that I can then pursue the things that I am personally passionate about.
I'm actually a customer of Tutanota, and their product has been pretty good in the last ~2 years I've used it.
Short of hosting your own email, I can't think of other email providers as secure as tutanota. They tick off a lot of boxes: open source, hosted in Germany, encrypted mailboxes, DKIM/DMARC support...
The only feature that might give people pause is that their search functionality isn't great. I have pretty low volume of emails to my personal inbox so it doesn't matter, but it might be frustrating to use as a work inbox.
- Why did you get into the field? What did you focus on at first?
I really liked video games. I remember some skills I learned as a very young child and teenager to do things like play Warcraft 3 online, install cracked versions of games, play on WoW private servers, host my own Minecraft server...
So then after being an aimless teenager I went to the equivalent of a community college to do "IT" at 20 years old. I learned things like windows/linux sysadmin, networking, printer management..
I never ended up finishing that degree though.
- What are you doing at your job? Is it everything you dreamed of and more?
I am basically an SRE. I was doing sysadmin for my company for about 2.5 years but put my foot down and decided that having work thrown over the fence to me by devs was not a good long term career plan.
Now I've been doing software development, but it's not user facing feature work. It's been a lot of improving application monitoring/debuggability/operability (distinct from usability).
It's not really my dream job, I've sort of figured out this company by now and I'd like to get a more difficult job.
- How did you break that first-job barrier?
I made personal friends with one of my teachers who basically coached me through how to get past the very difficult interview process in IT. Then I applied to internships on my university job board and got lucky eventually.
My boss later admitted to me that I wasn't hired because I had the best credentials, but mostly because I had a wide skill set and he had a gut feeling about my work ethic.
- What were you doing before this?
I worked at a moving company and played an obscene amount of Starcraft 2.
- Any tips for the rest of us?
Trying to get into IT is easier than trying to become a developer. You can start in help desk and work your way up, there are certifications for the specific technologies you will be working on (doing Cisco networking? get CCNA, doing cloud? Get the AWS solutions architect).
Once you get good at IT related skills, you'll actually pick up a lot of key software development skills without even realizing it. There is an embarrassing amount of time I have spent explaining things about git, SSH keys, layer 3 networking, HTTP, TLS... to very smart and qualified developers making triple my salary.
I did horrible end user support for my first little while. It's OK when you start out to be explaining to people how to connect to the VPN because they can't read a wiki page with screenshots, just make sure that you realize the job is a stepping stone for you to continue building skills. Once you get the job you cannot rest on your laurels.
I can at least anecdotally say that I am in a privileged position of being enormously happier after COVID-19 shut my workplace down and every employee was told to work from home. I associate this entirely with my commute being reduced to nothing. I hope working from home in IT becomes the future standard. Big expensive cities are not worth it.
To expand on the article. One thing I've noticed is that a lot of people in my age group (18-24 years old) are desperately trying to find happiness in "experiences" but finding emptiness due to social media. Often for them, it is not achieving something amazing like a long bicycle trip, or summiting a mountain that brings them joy, but they are thinking about the resulting Instagram photos instead.
Communication tools like Snapchat are a little bit healthier. Most people use Snapchat not to show off, but rather to regularly share their lives with a small group of friends.
Certain social media platforms like Instagram or TikTok seem entirely designed to try to get their users to pursue building a following and attach their happiness to that. Some unfortunate people end up having all the joy of traveling and meeting new people robbed from them as a result. They know that "experiences" are better for happiness than possessions, but somehow the result is a bastardized version the actual experience.
Knowing all of this, I am not taking a holier-than-thou position. I myself have accounts on several social media platforms, but make a conscious effort to not get drawn into their dopamine manipulating designs.
I used to maintain some linux remote desktops at my job for specific workflows of certain employees.
Wow, using xrdp was a complete gong show to manage. The most common bugs were in the assignment of displays on user logon, as well as a crappy, often buggy, login screen (though GUI logins for linux are fascinatingly weird pretty much universally.)
That being said, even in evaluating alternatives there wasn't anything much better. Linux remote desktop managers are just not that great yet.
As much as I hate Windows, I have to tip my hat to Microsoft's and say that their RDP implementation stands more than head and shoulders above any alternative remote desktop implementations I've tried.
Developers these days are getting much better at doing things over the command line. The trend of using GUIs for everything recently appears to be ebbing into a "right tool for the right job" mindset. Which is a very promising trend I've noticed. Hopefully remote workflows are entirely done through emulating terminals over SSH in a few years time :)
> Play video games that require intense focus for prolonged durations to perform well like Starcraft 2, CSGO, DOTA 2 helped me build mental endurance.
I used to be better than average at Starcraft 2. Probably peaked around top ~1.5% in the world wide rankings.
My thoughts on this is that the vast, VAST majority of people who play a competitive game like Starcraft 2 will not get any benefit from it. Following this advice blindly would be a disaster for 99% of people.
As a former high ranking master league player, I would mostly dedicate my time to memorizing and mindlessly practicing build orders. All responses I made in the game to my opponent were born from literally thousands of hours of practice and memorizing how certain interactions in the games went. 1 Marauder beats 1 Stalker. 2 Marauders beats 3 Stalkers. 2 Marauders loses to 4 Stalkers. For every single game interaction possible. I doubt an actual full formed thought ever crossed my mind during gameplay.
That's great for just zoning out and crushing noobs, but as a top 1.5% player I was absolutely TERRIBLE when compared to even low ranking professional players.
Why? Because they spent time watching replays to understand their opponents and to meticulously analyze their mistakes and think about solutions.
I didn't do any of that. I just memorized popular strategies and got good enough at pressing certain keys in certain orders that I was considered a "master". But I didn't master anything really. What does that say about the 98% of players who were lower ranked than me? What does that say about the larger player pool of Starcraft 2 which don't even compete, but rather play single player or arcade?
I want to make a critical correction to the quote that prompted me to reply.
> Practicing video games that require intense focus for prolonged durations to perform well like Starcraft 2, CSGO, DOTA 2 helped me build mental endurance.
The point I want to drive home is that concentrated, deliberate practice is the key to building mental fortitude.
I applied to a new grad production engineering job at Facebook recently.
I probably did about 80 leetcode puzzles, let's just assume that is 1 hour per puzzle. So 80 hours of leetcode puzzles.
In addition, 10 hours were spent reviewing my networking trivia (focused on layer 2/3/4 and HTTP).
The PE recruiter at the company also provided a study guide packet. I elected to sink about 10-20 hours into studying the Linux troubleshooting and finding some common questions off of Glassdoor to understand the type of things they will be asking.
Finally I tried to review some system design stuff (but as a juniour in this industry didn't really know how) Just followed some study guides from the system-design-primer on Github. Probably 5 hours dedicated to this.
Let's just round it to about 100 hours of prep in about a month and a half. While juggling a full time job as a devops engineer.
Afterwards I was rejected :)
Ultimately I wasn't surprised I was rejected. I was under the impression that I needed to pass all 7 of the interviews in order to get an offer (2 phone screens, 5 on site.) I know for sure I failed the system design interview question HARD. Like irredeemably hard. Regardless of my performances in my other interviews, I know I definitely did a good job in my network and OS interviews but it doesn't matter.
I do wish that companies would provide a feedback system. I would love to know where they thought I was weak so that I can improve upon that later. But there is nothing beyond "we decided to pursue other candidates."
Here's the crazy thing, though. It isn't even a question in my mind that I should try again if another FAANG recruiter reaches out to me. I will sink another 200 hours if necessary to pass their interviews because of what it represents to achieve getting hired at such a firm. There's prestige, salary, and quality of life that just can't be matched by any other tech companies.
I just remind myself that this world has 7.5 billion people on it, and as a programmer I have a better income than the VAST majority of them.
So, there are billions of people who would love to out-compete me and take my job. I need to make sure I'm always ahead of the curve so that the legions of CS grads being pumped out of universities don't replace me.
Due to how many North American cities are designed with urban sprawl and separation of living areas (suburbs) and productive areas (industrial/office zoning laws) it should actually be the norm to have a lengthy commute to work.
Hell, I live in a suburb with quite good public transport and it would take me almost 30 minutes to go to the grocery store by bus.
When I was a teenager and had no car, it took me about an hour to get to my minimum wage job by bus due to transfers. Once I could afford a car (because I lived with mum and dad and could save) that commute got cut to 20 minutes.
1. The consensus system you are talking about is Raft (https://raft.github.io/) and it's baked into Consul.
2. There are two ways to interpret parallel infrastructure. I will post my thoughts on both.
2a. Standing up a parallel Consul cluster. This is problematic because typically what people do with Consul is to put a Consul agent on every server (or pod) which registers its services for discovery to the Consul servers. When you make a parallel Consul cluster you need to also restart the Consul agents on every other service. They only mention Postgres in this blog post but there can potentially be a LOT of other servers registered to the Consul cluster.
2b. Standing up everything it takes to run Gitlab in parallel and then diverting traffic. Honestly sounds great. The reason a team wouldn't do this is due to either a) not having infrastructure code which allows for one-click deployment of whatever it takes to have Gitlab running. or b) it's actually pretty expensive to do if you're not Google or Amazon. The blog post mentions 255 clients (and 5 Consul servers). That's a lot of servers to rebuild!
Now, I would love to hear from anyone else who uses Consul, because I have my own thoughts on how they decided to handle the issue. I will focus my attention entirely on the Consul and not the Postgres portion.
The blog post mentions two limitations: 1) Reloading the configuration of the running service, which worked fine and did not drop connections, but the certificate settings are not included in the reloadable settings for our version of Consul. 2) Simultaneous restarts of various services, which worked, but our tools wouldn't allow us to do that with ALL of the nodes at once.
We don't need to reload. We can run a rolling systemctl restart whicn ansible is perfect for. The nice thing about this is that their stop-gap solution is to disable TLS verification. This means that servers with TLS verification ON in the meantime should be able to continue validating certs while other servers can have a rolling restart that disables TLS verification one server at a time. If we want to minimize downtime we would do every non-leader server in the cluster, then finally the leader, then every client in a serial manner. With 260 servers to deal with it would be slow but it shouldn't break Raft at any point. There is no reason for quorum to be broken. The gossip will still be communicated over TLS, just that some of the servers/clients wouldn't be validating the certs.
Then, we would follow exactly the same process for rolling out valid certificates with TLS validation turned back on. One non-leader server at a time, then the leader, then every client.
I could be missing some critical piece here, and it looks like the Gitlab team did run a lab test before making their change in prod. It's easy to miss a possibility when under pressure, and also easy for an online commentator like myself to think they are so much smarter. They still managed to get out of the crisis with no downtime and congrats to the operators who pulled it off!
It seems like this problem is solved by simply setting a sensible minimum in an autoscaling group. And not "everyone on Earth was abducted by aliens and stopped using the service" levels of minimum.
Say I'm an e-commerce site and on Black Friday I can see historically (or just make an educated guess if it's your first holiday sale) I get "n" requests per second to my service.
I'll set my autoscaling group the day before to be able to handle that "n" number of requests, with the ability to grow if my expectations are exceeded. If my expectations are not met, then my autoscaling group won't shrink. Then the day after the holiday sale, I can configure my autoscaling group to have a different minimum.
This solves the problem of balancing between capacity planning and saving money by not having idle resources running.
If you're the type of person who hates human intervention for running your operation, then fine. Put in a scheduled config change every year before a sale to change your autoscaling group size.
It's pretty rare to have enormous spikes in application usage without good reason. Such as video-game releases, holiday sales, startup openings, viral social media campaigns.
The book details the lessons over time that the author (an American chess prodigy) learned about how to improve at chess. It gets interesting when he applies these same lessons to become a world champion at combat Tai Chi.
The impact this had on me is how I apply myself to any competitive discipline to maximize learning. Not necessarily success, but the amount of knowledge gained from practice. This has the happy side effect of often leading to success.
But also I liked the stories of a child growing up in a competitive world, the amount of love and support he receives from his mentors, friends, and family. The ability to share this love with other students of chess and to produce profound insight into how people in general learn and react to hypercompetitive situations.
What is also great about the book is that it written in very accessible language. The reader does not need to be a scholar to understand the concepts Waitzkin is trying to express in his book.
For anyone not on the security stream of Debian packages for Bookworm, kernel version 6.1.0-42-amd64 is actually immune to copy.fail. Surprising that it looks to be immune to dirtyfrag. If you haven't already patched on the security stream, you can choose any kernel version that kept commit 2b8bbc64b5c2. I am thinking that the same commit might accidentally be keeping certain Debian 12 kernel versions safe from dirtyfrag as well.