I'm really "new" to x64 (I only migrated from 32-bit in 2020...) and the difference I noticed between x86-64-v1 and x86-64-v3 was only with video (with ffmpeg), audio (mp3/ogg/mp4...) and encryption; the rest remains practically the same.
Naively, I believe it might be more appropriate to have x86-64-v1 and x86-64-vN options only for specific software and leave the rest as x86-64-v1.
AVX seemed to give the biggest boost to things.
Regarding those who are making fun of Gentoo users, it really did make a bigger difference in the past, but with the refinement of compilers, the difference has diminished. Today, for me, who still uses Gentoo/CRUX for some specific tasks, what matters is the flexibility to enable or disable what I want in the software, and not so much the extra speed anymore.
As an example, currently I use -Os (x86-64-v1) for everything, and only for things related to video/sound/cryptography (I believe for things related to mathematics in general?) I use -O2 (x86-64-v3) with other flags to get a little more out of it.
Interestingly, in many cases -Os with -mtune=nocona generates faster binaries even though I'm only using hardware from Haswell to today's hardware (who can understand the reason for this?).
It's almost funny, not to mention sad, that their player/page has been changed, filling it with tons of JS that makes less powerful machines lag.
For a while now, I've been forced to change "watch?v=" to "/embed/" to watch something in 480p on an i3 Gen 4, where the same video, when downloaded, uses ~3% of the CPU.
However, unfortunately, it doesn't always work anymore.