First, the fact that I have to download the whole blockchain to use it. I'm not on a super fast connection, so that took like a day. The difficulty and expense of getting Monero was also an issue; I had to buy Bitcoins, then move Bitcoins to an exchange that would let me buy Monero, because the exchange I could buy Bitcoin on didn't work with Monero (due to the perception that it's only used for criminal activity). At every step, there's a transaction fee, and that fee isn't entirely transparent up front, so it's harder to estimate what the final price (in fiat currency) will be.

At the tiem I was trying to use it, there weren't any user-friendly wallets, and I don't think there was any capability to use it from a mobile phone; that makes it more difficult to use than other crypto.

I'm not sure how well it plays with Tails of Qubes; I never got far enough to give it a shot.

I'm not saying that any of these thigns are bad, but they do make it harder for a typical person to start using, and until more regular people are using privacy-focused crypto and operating systems, they're always going to have the appearance of being used for crime only.