I think Linux phones have huge potential going forward. Personally, I've noticed that my usage patterns haven't really changed in at least a decade. I use a fairly small number of apps like calendar, email, chat, media player, etc. The way these apps work hasn't really changed, and if I could get those working well on a Linux phone then I really wouldn't miss Android one bit.
As a side note, it's really depressing to see how hardware keeps getting faster, but software bloat negates all the gains. If we could have an ecosystem of open source apps that are optimized for performance, then we could have phones that last for many years without having to keep replacing them. Imagine if with every update your software kept getting faster instead of slower.