So I want to be clear. This work is very much in the "fantasy" stage. I haven't ran this at scale, and there is a lot I would do before I blindly rolled something like this out. As far performance impact, it obviously would depend on how it's implemented however I think a reasonable amount of "tracer packets" being sent out every 15 seconds or so to each hop shouldn't be too disruptive to the network, or to the host machine issuing the request. In other words if you take the `ptrace(2)` concern out of the equation (this is the big one in my mind) its going to be negligible.
> And, the rootkit question, how would eBPF notice you doing this?
I suppose it would depend on what your strategy with eBPF is and where you were looking. Reminding yourself that the pidfd_getfd and pidfd_open functions are system calls, I think any modern Linux auditing system would see something like this "a mile away".
In other words, I don't see somebody using this tactic to steal FDs in production without quickly alerting most security systems. I suspect there would be ways of hiding this however... https://github.com/krisnova/boopkit...
I often attribute the vast majority of my happiness and success back to psychedelics, trauma, and my experience living an unconventional and eccentric life.
I could fill pages with the anecdotes, specific examples, and my personal speculation on why this is but i can just gloss over the details and skip to the takeaway: LSD has made my life net better.
Another point I am trying to drive home is that this is what I would consider an “incomplete” or “immature” take. For example the authors anxiety is likely stemming from some organic experiences either traumatic, economic, physiological, or psychological — getting to the bottom of those and surfacing the patterns themselves is one of the great benefits of this chemical. I’d love to see this author examine their experience and ask where the anxiety is coming from.
You have to face your fears and anxiety before you can understand them — before you can conquer them.
I’m just glad this forum is finally talking about this topic. More psychedelics here please.
This only works if you’re a bro who is chummy with leadership. Honestly this is one of the worst pieces of advice for a marginalized person. If you show any indication of leaving as a marginalized person you are extremely likely to be pegged as “a flight risk” or “unstable” or “frantic” and your job will likely be eliminated before you actually are ready to roll.
In the U.S. this likely means losing your healthcare and missing your children’s tuition payments — and even your housing. Your visa if you’re immigrating, etc.
The only people who have enough psychological security to do this are the ones who don’t actually depend on their jobs for maintaining their current standard of living.
In a perfect world this would be great advice for everyone, however given the political and economic culture of the U.S. this is pretty horrible advice for a marginalized person. Our labor laws don’t support this behavior for a reason — and if “push comes to shove” every corporation in Silicon Valley will air on the side of modern labor laws (or lack there of).
Wow. I thought this was going to be a big rant piece on why we should all go back to dynamic linking again and I was so ready to start my Monday off with a spicy take.
But yes we should all also be adding links in our writing and publishing on our own platforms.
Women and marginalized people who change jobs: Flakey and incapable. Unable to handle a job. Something must be wrong. Clearly a sign of caution to be taken as a reason not to work with them.
Men who change jobs: literally articles inventing new vernacular stemming from the mental gymnastics required to justify the hypocrisy — men aren’t incapable because they change jobs — they are prodigy — men aren’t untrustworthy for changing jobs — they are taking nonlinear career paths because of the uncertainty in the market
Fortunately, I know I can always trust the charming comments on this enchanting website to let me know how the general public is feeling about my systems neurosis.
Excellent. Most people and resources sited on this Machiavellian capitalist mega-bro site often disagree with me. I take it as a compliment. It’s a positive sign I’m on to something sustainable and worth investing my time into. Thanks for helping me smile today.
We debugged a lot of the behavior using hackyderm.io as an HTTP facade for hachyderm.io with independent TLS termination.
I was able to get a single tweet in using hackyderm.io and my account is now “shadow banned”. Meaning I can view my tweet, but nobody else can. Which is exactly what Twitter has promised not to do, and what we use as a tactic on Reddit to keep shitty content at bay. It means that from the posters perspective everything looks “fine” and their content is just not getting any engagement.
A Microsoft SAW is a demonic artifact stemming from when Satan himself forged a computer a casts it onto the surface of the earth and into the hands of mankind. It’s the best worst way to access production. It’s how anyone who has ever been on call for Azure accesses production and it’s the last thing you see before Cerberus herself begins chanting “Lasciate ogne speranza, voi ch’intrate” and welcomes you to hell for all eternity.
Hi. I wrote the post. Additionally I am responsible for operating Hachyderm (Ruby on Rails) and GitHub (Ruby on rails) for both my free-time and my day job.
I can say with certainty that Ruby specifically was not the bottleneck in our case. I do think that the rails paradigm can often lead to interdependent systems. We see this at GitHub and we also see this in Mastodon. Service A will do reads/writes against the same tables in the database that Service B also does. When service A is moved to an isolation zone, it can still impact Service B's performance.
In other words, I think any stateful framework with the flexibility that Ruby on Rails encourages bad behavior that can contribute to a noisy neighbor problem.
The point I am trying to drive home is that I agree. I can confidently say that Ruby on Rails is not the culprit in our case. To be honest I just ignore anyone who is quick to point fingers and assign blame either technically or personally.
Sorry hacker news got you down. If it helps my family and I are making Sunday morning pancakes with my puppy Björn today and we are all wishing you the best day ever.
Hi, I made the decision not to replace the drives. I also wrote the article, and am the admin of Hachyderm.
So to be clear, we did try to "offline" a drive from the ZFS pool just to see if this was a viable path. The ZFS pool was set up a few years ago and has gone through a few iterations of disks. The mirrors were unbalanced. We had pairs of drives of one manufacturer/speed mirrored with pairs of drives from another manufacturer/speed. We know this configuration was wrong, again we didn't intend for our little home lab to turn into a small production service.
I think after spending a few hours trying to "offline" the disk, and then repairing the already brittle ZFS configuration to getting the database/media store back to a "really broken and slow but still technically working" state we just decided to pull the plug and move to Hetzner. Offlining the disk caused even more cascading failures and took about 30 minutes just for the software. We could have technically shut down production to try without the database running on it, but at that point we decided to just get out of the basement.
If it would have been as easy as popping a disk in/out of the R630 (like one would imagine) we would have certainly done that.
To be honest I am still very interested in performing more analysis on ZFS on a 6.0.8 Linux kernel. I am not convinced ZFS didn't have more to do with our problems than we think. I will likely do a follow up article on benchmarking the old disks with and without ZFS in the future.
I’d like to get involved with contributing to the prossimo project like the blog suggested, however there doesn’t seem to be any more information anywhere on memorysafety.org to do so.
So I want to be clear. This work is very much in the "fantasy" stage. I haven't ran this at scale, and there is a lot I would do before I blindly rolled something like this out. As far performance impact, it obviously would depend on how it's implemented however I think a reasonable amount of "tracer packets" being sent out every 15 seconds or so to each hop shouldn't be too disruptive to the network, or to the host machine issuing the request. In other words if you take the `ptrace(2)` concern out of the equation (this is the big one in my mind) its going to be negligible.
> And, the rootkit question, how would eBPF notice you doing this?
I suppose it would depend on what your strategy with eBPF is and where you were looking. Reminding yourself that the pidfd_getfd and pidfd_open functions are system calls, I think any modern Linux auditing system would see something like this "a mile away".
In other words, I don't see somebody using this tactic to steal FDs in production without quickly alerting most security systems. I suspect there would be ways of hiding this however... https://github.com/krisnova/boopkit...