Updating display while WiFi connected breaks WifiClient::connect()?

[rogueeve575]

So I modified the stock firmware demo ("Watchy" library) to replace the About Watchy menu item with a screen that's a little TV remote control which uses WiFi to connect to a simple server I wrote in C running on the computer that's connected to my TV, and synthesize a few common media control keys as selected from a menu on the watch. It's using a very simple protocol over raw TCP that just requires sending a modicum of authorization (sending a "random" uint32_t followed by another uint32_t hashed from the combination of the password and the first number), followed by a byte representing the key to be pressed, or "<" followed by a special sequence (such as <VOL+ or <BKSP).

It works great and is a lot of fun but one problem I can't resolve is that I could only to get it to work reliably if for every time the user asks to send a key I connect to wifi, connect to the server, send my packets and then shut down the wifi again, which is very slow. I tried to get it to leave the wifi connected for 12.5 seconds afterwards and re-use the connection for subsequent keypresses before going back into deep-sleep mode, but when I do this, about 70%-80% of the time WiFi.status() still returns WL_CONNECTED, but client.connect() hangs for a few seconds and then reports failure.

I created my own loop similar to the stock firmware's "fast menu" mode which it stays in for ~10 seconds on entering the "app" or after the first button press which wakes it back up from deep sleep so I know that other parts of the firmware shouldn't be messing anything up.

During debug I tried wrapping my entire connect/send packet/disconnect code in a big for loop and connecting/disconnecting (to the server, not the Wi-Fi) 50 times after wifi is brought up, and this works as expected 100% of the time.

I eventually traced it don to my showRemote function where if I comment out the display.display(), it doesn't exhibit the issue anymore (but of course you can only see what's going on through the debug serial interface).

I'm not too familiar with Arduino libraries as I usually prefer writing directly to the chip and am just recently getting into ESP32 development, however I have plenty of programming and embedded development experience on older devices such as AVRs.

Bear with me, I'll try to be thorough while hopefully not dumping too much non-relevant code...First, here is the main loop which the watch sits in for a while after activity before returning back to handleButtonPress and the normal Watchy code and going back to deep-sleep:

    static int getButton()
    {
            static int buttons[] = { MENU_BTN_PIN, BACK_BTN_PIN, UP_BTN_PIN, DOWN_BTN_PIN, -1 };
            for(int i=0;buttons[i] != -1;i++)
            {
                    if (digitalRead(buttons[i]))
                            return buttons[i];
            }
            
            return 0;
    }

    void Watchy::enterRemote()
    {
            debug("Entering TV Remote control mode...");
             
            guiState = TV_REMOTE_STATE;
            remoteState = REMOTE_READY;
            remoteErrorCode = -1;
            remoteMenuIndex = 0;
            remoteSentKey = -1;
            remoteSentSeq = 0;
            remoteWifiConnected = false;
            remoteLastMenuIndex = -1;
             
            showRemote();

            pinMode(DOWN_BTN_PIN, INPUT);
            pinMode(UP_BTN_PIN, INPUT);
            pinMode(MENU_BTN_PIN, INPUT);
            pinMode(BACK_BTN_PIN, INPUT);
            
            debug("waiting for button release...");
            while(getButton()) { }
            
            debug("entering spinloop...");
            unsigned long expiry = millis() + 12500;
            while(millis() < expiry)
            {
                    int btn = getButton();
                    if (btn)
                    {
                            expiry = millis() + 12500;
                            if (btn == BACK_BTN_PIN) break;
                            else handleRemoteButton(btn);
                    }
            }
            
            debug("leaving spinloop");
            remoteNotifyExitFastMode();     // shutdown wifi if still active
    }

...here is the relevant showRemote function which draws the menu:

#define REMOTE_LAST_MENU_ENTRY			5
static const struct {
	const char *menuEntry, *cmd;
} remoteMenu[] = {
	{ "Play/Pause", " " },
	{ "Subs", "s" },
	{ "Vol Dn", "<V-" },
	{ "Vol Up", "<V+" },
	{ "<- Back", "<LT" },
	{ "Fwd ->", "<RT" },
	{ NULL, NULL }
};

    void Watchy::showRemote(bool partialRefresh, uint32_t drawFlags)
    {
            display.setFullWindow();
            
            if (drawFlags & REMOTE_DRAW_CLEAR)
                    display.fillScreen(GxEPD_WHITE);
            
            display.setFont(&FreeMonoBold9pt7b);
            display.setTextColor(GxEPD_BLACK);
            display.setCursor(0, 40);

            // ...bunch of calls to display.println(), fillRect() and setCursor...

            display.display(partialRefresh);
    }

handleRemoteButton is just a switch statement; here's the relevant bit (the other cases just move the menu selection):

    void Watchy::handleRemoteButton(int btn)
    {
            debug("handleRemoteButton %d", btn);
    
            ...

            case MENU_BTN_PIN:
            {
                    vibMotor(10, 100/10);
                    remoteSentKey = remoteMenu[remoteMenuIndex].cmd[0];
                    remoteSentSeq++;
                    remoteState = REMOTE_SENDING;
                    showRemote(true, REMOTE_DRAW_STATUS);
                    
                    debug("Selected index %d, key %02x", remoteMenuIndex, remoteSentKey);
                    
                    remoteSendCommand(remoteMenu[remoteMenuIndex].cmd);
                    showRemote(true, REMOTE_DRAW_FULL_MENU);
            }
            break;
            
            ...

...and finally here is the meat of it all that gets called with the key to send once the wearer selects a menu item:

static uint32_t calcauth(uint32_t challenge)
{
	uint8_t st[9+4] = {
		'm', 'y', 'c', 'a', 't', '4', 's', '-', '-',
		(uint8_t)((challenge >> 24) & 0xff),
		(uint8_t)((challenge >> 16) & 0xff),
		(uint8_t)((challenge >> 8) & 0xff),
		(uint8_t)((challenge & 0xff))
	};
	
	uint32_t sum = 5381;
	for(int i=0;i<sizeof(st);i++)
		sum += ((sum * 33) + (st[i] ^ 0x7e));
	
	return sum;
}


void Watchy::remoteSendCommand(const char *cmd)
{
bool leaveConnected = false;
bool reusedWifi = false;

	debug("");
	debug("remoteSendCommand '%s'", cmd);

retry: ;
	if (WiFi.status() != WL_CONNECTED)
	{
		debug("Connecting to Wi-Fi...");
		if (!connectWiFi())
		{
			debug("Wifi Connect Failed!");
			remoteState = REMOTE_ERROR;
			remoteErrorCode = 1;
			return;
		}
		
		remoteWifiConnected = true;
		reusedWifi = false;
	}
	else
	{
		debug("Re-using existing Wi-Fi connection.");
		reusedWifi = true;
	}
	
	#ifdef SERDEBUG
		Serial.print("obtained IP address ");
		IPAddress ip = WiFi.localIP();
		Serial.println(ip);
	#endif
	
	WiFiClient client;
	IPAddress addr(10, 0, 0, 45);
	#define PORT		1985
	
	if (reusedWifi)
		;//client.setTimeout(100);
	
	debug("Connecting to server...");

	// [**] THIS IS THE CALL WHICH UPDATING THE DISPLAY SEEMS TO SCREW UP
	if (client.connect(addr, PORT))
	{
		debug("Sending auth packet...");
		
		uint32_t challenge = ++remoteChallengeSeq;
		uint32_t response = calcauth(challenge);
		
		uint8_t authpkt[8];
		uint32_t *ptr = (uint32_t *)&authpkt[0];
		*(ptr++) = htonl(challenge);
		*ptr = htonl(response);
		
		debug("Sending challenge/resp %08x:%08x", challenge, response);
		client.write(authpkt, sizeof(authpkt));
		
		debug("Sending command.");
		client.write(cmd, strlen(cmd));
		client.flush();
		
		debug("Waiting for response...");
		
		int timeout = 1000;
		while(!client.available()) {
			delay(1);
			if (--timeout <= 0) {
				debug("Timeout waiting for response");
				remoteState = REMOTE_ERROR;
				remoteErrorCode = 30;
				goto shutdown;
			}
		}
		
		uint8_t resp[2] = {0};
		int sz = client.read(resp, 2);
		if (sz != 2)
		{
			debug("Wrong size resp %d", sz);
			remoteState = REMOTE_ERROR;
			remoteErrorCode = 40;
			goto shutdown;
		}
		
		if (resp[0] != 'O' || resp[1] != 'K') {
			debug("Expected 'OK', got resp %02x.%02x", resp[0], resp[1]);
			remoteState = REMOTE_ERROR;
			remoteErrorCode = 50;
			goto shutdown;
		}
		
		debug("Server confirms OK!");
		
		// temphack: bug of some kind in Arduino libs or hardware bug:
		//	it seems that calling display.display() can break WiFi if you leave it connected -
		//	the wifi will still report connected, but client.connect() may fail 60% of the time next time if
		//	we leave it connected across a display refresh.
		//leaveConnected = true;
		remoteState = REMOTE_SUCCESS;
	}
	else
	{
		debug("client.connect() failed!");
		
		if (reusedWifi)
		{
			debug("wifi was re-used, trying a re-connect");
			remoteNotifyExitFastMode();
			goto retry;
		}
		
		remoteState = REMOTE_ERROR;
		remoteErrorCode = 2;
	}
	
	client.stop();
	
	if (!leaveConnected)
	{
shutdown: ;
		remoteNotifyExitFastMode();
	}
	else {
		debug("Leaving Wi-Fi connected...");
	}
}

void Watchy::remoteNotifyExitFastMode()
{
	if (remoteWifiConnected || WiFi.status() == WL_CONNECTED)
	{
		debug("Shutting down radios...");
		
		int save = remoteState;
		remoteState = REMOTE_SHUTDOWN;
		showRemote(true, REMOTE_DRAW_STATUS);
	
		WiFi.mode(WIFI_OFF);
		btStop();
		remoteWifiConnected = false;
	
		remoteState = (save == REMOTE_ERROR) ? save : REMOTE_READY;
		showRemote(true, REMOTE_DRAW_STATUS);
	}
}

As I said there is no issue if I simply call remoteSendCommand repeatedly while taking out the parts that update the screen. And as you can see this screen takes over the Watchy while it's in the spin loop so other parts of the library shouldn't be running. Also debug() is just a macro for Serial.println(), but only if SERDEBUG is defined; otherwise it's just a no-op. So there is no serial communication actually going on in production. That didn't seem to affect it either way though as I was able to enable debugging and send multiple commands if I disabled the call to display.display().

So I'm not sure if this is some kind of hardware bug or if the GxEPD2 library's display.display() is doing something that breaks Wi-Fi, maybe something to do with the GPIO's, SPI, or sleep modes. I think it's interesting that it still reports WL_CONNECTED (always), and that it occasionally does reconnect successfully for a 2nd button press with leaveConnected set to true as it's supposed to be (but not very often).

I also noted that none of the few Wi-Fi features in the stock firmware ever leave it connected, so they wouldn't run into this bug/problem.

And lastly I don't know if this is normal or not or is related at all but I did notice that the range is not very good compared to other devices in my house. It will often fail to connect if I'm more than one room away from the room with the AP. My wifi signal is strong though and other devices have no issue. I'm using the aluminum Gameboy case so I don't know if that might weaken the signal. And of course I do realize that the 3D antenna is really tiny but I expected better range than that, so I don't know if that has anything to do with my issue. In testing I am sitting right next to the router and it always connects to wifi fine, just is unreliable about actually connecting to the server if I update the e-paper between bringing up wifi and calling WiFiClient::connect().

[ZinggJM]

Hi,

I was pointed to this issue by e-mail. I don't expect e-mails for issues with my libraries, but decided to give a short answer.

This issue may be related to:

void Watchy::displayBusyCallback(const void *) {
  gpio_wakeup_enable((gpio_num_t)DISPLAY_BUSY, GPIO_INTR_LOW_LEVEL);
  esp_sleep_enable_gpio_wakeup();
  esp_light_sleep_start();
}

For further analysis you could comment out the code of this method to see if it helps.
You could try to let ESP32 auto-connect, if this exists (I use it on ESP8266).