Lots of projects with good security track records host key servers in a "member's house". That said, the F-Droid buildserver is not hosted in anyone's house.
There are two key concepts at play here: "least authority" and "infrastructure as code". The buildserver host is sensitive security-wise, but easy to set up an instance. We have multiple instances running, and spin up new ones from time to time. For production infrastructure, there should only be enough people with access to it as are needed to maintain it. No more. If a sysadmin goes rogue, we can always just spin up a new instance elsewhere with a new maintainer.