I don't think the privilege is just about being handled things easily, it's also about being able to reach such positions, no matter what the upbringing you had.
A lot of people are going to work as hard or harder than you, but because they chose a different career or because they don't have your set of skills, they'll never reach what you got and that is a huge privilege.
I used both and even though their purpose is very similar (have an "immutable" system tree that you create and switch into it), the day to day is very different.
Silverblue is still pretty much imperative, you install/remove RPM packages and that's it, you use Flatpak for everything else. NixOS you have to describe your entire system in a programming language. NixOS gives you so much more freedom to do what you want, but you have to work for it, learn a language, learn it's constructs, etc.
I enjoy both though, feels like the right direction to go, you just need to choose how you want to interact with your OS.
A lot of people are going to work as hard or harder than you, but because they chose a different career or because they don't have your set of skills, they'll never reach what you got and that is a huge privilege.