I got an ESP32 WROOM 32U board that I intent to use it as the flight controller which will keep the drone within its flight envelope and do the maneuvering upon receiving commands from the phone.

I find the phone appealing because I want try to make it somewhat autonomous, like im RTS games where you give a unit a command to go somewhere and it figures the path out by itself and avoids obstacles without direct input. The phone has quite a lot of processing power and sensors and IMHO doing it on board will be more interesting. Also, I'm not sure that the connection will be always stable and high bitrate make the drone a thin client.

Still I think I'd just have the phone run an app that talks to a microcontroller by Wifi, even if it is strapped to the drone all the time.

I haven't tried it yet but as it appears that will be my choice too. Other options appear to be bluetooth or ethernet tethering over USB.