Unless you audit the entire codebase prior to a manual build, from a machine you know hasnt been compromised with a key you know hasnt leaked, how is a manual build different to CI/CD securitywise?
This is called two factor auth. SMS is not considered a secure method of transport (mostly due to porting). It only solves opportunistic password compromise via password dumps.
I wasn't aware that people hated it. I very rarely struggle through it (learning curve related, probably) but the objective behavior is something I wish Linux tools would move towards.