Smart switches are programmable, and can easily configure smart switches and lights. You can get a touch screen interface to home assistant, and do all of that on it, embed it on the wall. It doesn't need to be an app on your phone.

Voice is definitely easier and more convenient, with HA being more configurable and difficult.

There are always going to be trade-offs in life, but you're definitely getting convenience in exchange for privacy here