The side port is only for charging the battery or powering the unit via USB. For programming, you have to use the port on the LOLIN D1 mini, inside the completed unit.
That is, unless you modify your ESPboy to connect the USB data lines to the side port, like I did: