The identifier is unavoidable for push notifications to work. It needs to know which phone to send it after all, even if it doesn't use Google's services, it would still need a way to know which device has new messages when it checks in. If it's not a phone number it's gonna be some other kind of ID. Messages need a recipient.

Also, Signal's goal is protecting conversations for the normies, not be bulletproof to run the next Silk Road at the cost of usability. Signal wants to upgrade people's SMS messaging and make encryption the norm, you have to make some sacrifices for that. Phone numbers were a deliberate decision so that people can just install Signal and start using E2E texting immediately.

If you want something really private you should be using Tor or I2P based solutions because it's the only system that can reasonably hide both source and destination completely. Signal have your phone number and IP address after all. They could track your every movements.

Most people don't need protection against who they talk to, they want privacy of their conversations and their content. Solutions with perfect anonymity between users are hard to understand and use for the average person who's the target audience of Signal.