If you can hard wire an ESP32 via Ethernet that would probably be best. You could also get better latency with an ESP32 over WiFi. However, a ZigBee button should be pretty low latency already.
Anecdotally I get perceptibly instant response from an ESP32 over WiFi.