Whatsapp only e2e encrypts message contents. The only thing Signal knows about you at any given time is the time of account creation and the date of your account’s last connection to Signal servers. That's tied to your phone number. They don't know who you chat with, the contents of those messages, your phone contacts, anything.
I'd get a chuckle out of comparing that with the privacy of Whatsapp.