If you're always using a VPN, that's not necessarily a privacy threat on your VPN'd device, but any other device on the network that doesn't have a VPN could be exposing itself to the ISP.
Also, you're at the mercy of whatever firmware updates your ISP issues for the router. Hopefully they remember to support your box when the next CVE is discovered...
We are forced to keep an ISP router/gateway combo in our home because it has certificates necessary to authenticate our subscription. However, behind that router we have the "real" router with settings and firmware updates that we control. The ISP router is just a hop between our router and the outside world. Everything on our network only connects to the router we control.