Current implementation uses a server, though you can fairly easily host your own, or find another server to your liking.
There is significant work being done to get matrix working p2p too, though. There is a demo of it, but it's not production ready. See https://p2p.riot.im/
There actually has been work done on it, partially by me.
Unfortunately it's simply an unsolved problem to properly federate the 3pid mappings. The long and short of it is that you need to find some way to prove to other identity servers that you properly verified a 3pid. That's theoretically possible with e-mail and DKIM, except there are too many e-mail servers that don't implement it correctly.
For phone numbers it's even worse.
[edit]
I forgot: There is a working implementation of federated identity servers for e-mail, but that requires you to run an identity server on the domain of the e-mail address that you're trying to add as 3pid. That means it works fine if you host your own e-mail but it doesn't work for the gmails and hotmails of this world.