My group does discord, we play on VTTs so built in voice chat was a requirement. You really don't need to engage with any features you don't want/need to. If you're the only DM you can be the server owner and leave everyone else a default user.
For a more FOSS answer I think Matrix would cover basically everything that discord does. I haven't used it that extensively but it seems to be the best discord alternative when dealing with larger groups where you may want several text channels.