How secure something is an spectrum. Sure self hosted matrix is a lot safer than sending your messages through meta servers for example. It's about what is the threat levels of what one is doing. Total tinfoiling like writing your own quantum proof multi encryption ciphers and sending that over an tamper proof usb stick with self destruct mechanism by a carrier pridgeon is not necessary or practical for average people who just want privacy, but for critical government applications and especially the military it might be. That is what we are talking about here.