I ran the Matrix server for HOPE and have fairly extensive experience administrating Matrix. For HOPE, we didn't even begin to approach anything that looked like capacity; any resources that were maxing out were either due to the DoS attack we were enduring (and made numerous changes to mitigate) during the conf, or applications in different threads that matrix metrics were picking up on (bug filed, will fixed in new versions).
Even so, we could easily - without a sweat - get 2000 people on a server. I wasn't even using more workers than we had cores. Make that a bigger box and you get a few thousand more. Then use a few different servers that are federated with each other for DEFCON and you have instant scale.
The matrix.org people run the matrix.org homeserver for over ~200k active users. You can see an outdated version of the stats here:
https://matrix.org/faq/#what-is-the-current-project-status
and they've said their active userbase has grown 300% in the past year.
The original plan for HOPE was to run multiple homeservers, but we didn't even begin to need to.