[SYS] Configuration portal for Ethernet board and HA link to config (#1761)

* [SYS] Configuration portal for Ethernet board

Enable Wifi Manager as a configuration portal for Ethernet boards, also leverage the Wifi Access point as a fallback connectivity method

Add link to the WebUI from HA and reduce key length

* Remove unnecessary environment

Now that one environment cans support wifi and ethernet, merge the 2

.github/workflows/build.yml

@@ -76,7 +76,6 @@ jobs:
           - "airm2m_core_esp32c3"
           - "lolin_c3_mini"
           - "thingpulse-espgateway"
-          - "thingpulse-espgateway-ethernet"
     runs-on: ubuntu-latest
     name: Build with PlatformIO
     steps:

docs/upload/portal.md

@@ -1,15 +1,19 @@
-# WiFi and MQTT configuration
+# Wifi and MQTT configuration
 
-## WiFi and MQTT configuration portal for WiFi boards
+## Configuration portal
 
 Once loaded into your ESP, and if you don't use the manual configuration, you have to set your network parameters with WiFi Manager portal.
 
-From your smartphone, search for  OpenMQTTGateway or OMG WiFi network and connect to it with the `WifiManager_password` you have defined (the default password is **"your_password"**), or if you are using a device sold by Theengs or the macro `-DWM_PWD_FROM_MAC=true`, the password will be the last eight digits of the device MAC Address with upper case.
+From your smartphone, search for  OpenMQTTGateway or OMG_ WiFi network and connect to it with the `WifiManager_password` you have defined (the default password is **"your_password"**), or if you are using a device sold by Theengs or the macro `-DWM_PWD_FROM_MAC=true`, the password will be the last eight digits of the device MAC Address with upper case.
 Example, the password would be `CCDDEEFF` for a MAC Address `AABBCCDDEEFF`. 
-For Theengs devices, the MAC Address can be found on the device sticker, and for all the devices, it is printed on the serial monitor logs.
+For the Theengs Plug, the MAC Address can be found on the device sticker, and for all the devices, it is printed on the serial monitor logs.
 
 Once connected to the WiFi, a web page should appear. On Android, you may also have a popup asking you if you want to connect to it without an internet connection. Answer yes always/all the time. If the web page doesn't appear, click on the WiFi Access Point and "Manage router".
 
+::: tip
+For boards with an ethernet port you can also access to the Wifi Manager portal through the LAN board IP address.
+:::
+
 ![WiFi manager menu](../img/OpenMQTTGateway_Wifi_Manager_menu.png)
 
 * Click on Configure WiFi
@@ -18,6 +22,11 @@ Once connected to the WiFi, a web page should appear. On Android, you may also h
 
 * Select your WiFi
 * Set your WiFi password 
+
+::: note
+If the board is connected by ethernet, the Wifi and password can be empty. If you fill them the board will use this wifi as a fallback connectivity method.
+:::
+
 * Set your MQTT Server IP
 * Set your MQTT Server Port (default: 1883)
 * Set the MQTT secure connection box to select whether or not the connection should be secure
@@ -37,12 +46,11 @@ The ESP restart and connect to your network. Note that your credentials are save
 Once done the gateway should connect to your network and your broker, you should see it into the broker in the form of the following messages:
 ```
 home/OpenMQTTGateway/LWT Online 
-home/OpenMQTTGateway/version
 ```
 
-Note that the web portal appears only on first boot, if you want to configure again the setting you can do a long press on TRIGGER_GPIO or [erase the settings](../use/gateway.md#erase-the-esp-settings).
+Note that the web portal appears only on first boot, if you want to configure again the setting you can do a long press on TRIGGER_GPIO, [erase the settings](../use/gateway.md#erase-the-esp-settings) or Reset the configuration from the WebUI.
 
-## MQTT and network configuration for Ethernet and WiFi boards
+## Build time configuration
 
 You can configure your MQTT server credentials and network configuration before building the application. It can be done either in Arduino or in Platformio IDE thought the User_config.h file. Note that with Platformio IDE you can also set your credentials into the platformio.ini file or an *_env.ini file, here is an example with the Olimex ESP32 gateway:
 

environments.ini

@@ -1613,34 +1613,6 @@ build_flags =
   '-DZgatewayBT="BT"'
   '-DLED_SEND_RECEIVE=2'
   '-DLED_SEND_RECEIVE_ON=0'
-  '-DpubBLEServiceUUID=true'
-  '-DGateway_Name="OMG_THINGPULSE_BLE"'
-  '-DANEOPIX_IND_DATA_GPIO=32'
-  '-DANEOPIX_IND_NUM_LEDS=4'
-  '-DANEOPIX_INFO_LED=0'
-  '-DANEOPIX_SEND_RECEIVE_LED=1'
-  '-DANEOPIX_ERROR_LED=2'
-  '-DANEOPIX_BRIGHTNESS=255'
-  '-DRGB_INDICATORS=true'
-;  '-DsimplePublishing=true'
-custom_description = BLE Gateway using Wifi
-custom_hardware = ThingPulse gateway single ESP32
-
-[env:thingpulse-espgateway-ethernet]
-platform = ${com.esp32_platform}
-board = esp32dev
-board_build.partitions = min_spiffs.csv
-lib_deps =
-  ${com-esp32.lib_deps}
-  ${libraries.ble}
-  ${libraries.adafruit_neopixel}
-  ${libraries.decoder}
-build_flags =
-  ${com-esp32.build_flags}
-  '-DZgatewayBT="BT"'
-  '-DLED_SEND_RECEIVE=2'
-  '-DLED_SEND_RECEIVE_ON=0'
-  '-DpubBLEServiceUUID=true'
   '-DGateway_Name="OMG_THINGPULSE_ETH_BLE"'
   '-DESP32_ETHERNET=true'
   '-DETH_PHY_TYPE=ETH_PHY_LAN8720'
@@ -1657,6 +1629,5 @@ build_flags =
   '-DANEOPIX_BRIGHTNESS=255'
   '-DRGB_INDICATORS=true'
 ;  '-DsimplePublishing=true'
-custom_description = BLE Gateway using ethernet, requires PIO configuration
+custom_description = BLE Gateway using ethernet or wifi with external antenna
 custom_hardware = ThingPulse gateway single ESP32
-

main/User_config.h

@@ -140,9 +140,8 @@ const byte mac[] = {0xDE, 0xED, 0xBA, 0xFE, 0x54, 0x95}; //W5100 ethernet shield
 
 /*-------------DEFINE YOUR ADVANCED NETWORK PARAMETERS BELOW----------------*/
 //#define MDNS_SD //uncomment if you  want to use mDNS for discovering automatically your IP server, please note that mDNS with ESP32 can cause the BLE to not work
-#define maxConnectionRetry     10 //maximum MQTT connection attempts before going to wifimanager setup if never connected once
+#define maxConnectionRetryNetwork 5 //maximum Wifi connection attempts with existing credential at start (used to bypass ESP32 issue on wifi connect)
-#define maxConnectionRetryWifi 5 //maximum Wifi connection attempts with existing credential at start (used to bypass ESP32 issue on wifi connect)
+#define maxRetryWatchDog          11 //maximum Wifi or MQTT re-connection attempts before restarting
-#define maxRetryWatchDog       11 //maximum Wifi or MQTT re-connection attempts before restarting
 
 //set minimum quality of signal so it ignores AP's under that quality
 #define MinimumWifiSignalQuality 8

main/ZmqttDiscovery.ino

@@ -346,7 +346,7 @@ void createDiscovery(const char* sensor_type,
 
   StaticJsonDocument<JSON_MSG_BUFFER> jsonDeviceBuffer;
   JsonObject device = jsonDeviceBuffer.to<JsonObject>();
-  JsonArray identifiers = device.createNestedArray("identifiers");
+  JsonArray identifiers = device.createNestedArray("ids");
 
   if (gateway_entity) {
     //device representing the board
@@ -354,17 +354,25 @@ void createDiscovery(const char* sensor_type,
 #  ifndef GATEWAY_MODEL
     String model = "";
     serializeJson(modules, model);
-    device["model"] = model;
+    device["mdl"] = model;
 #  else
-    device["model"] = GATEWAY_MODEL;
+    device["mdl"] = GATEWAY_MODEL;
 #  endif
-    device["manufacturer"] = GATEWAY_MANUFACTURER;
+    device["mf"] = GATEWAY_MANUFACTURER;
-    device["sw_version"] = OMG_VERSION;
+    if (ethConnected) {
+#  ifdef ESP32_ETHERNET
+      device["cu"] = String("http://") + String(ETH.localIP().toString()) + String("/"); //configuration_url
+#  endif
+    } else {
+      device["cu"] = String("http://") + String(WiFi.localIP().toString()) + String("/"); //configuration_url
+    }
+
+    device["sw"] = OMG_VERSION;
     identifiers.add(String(getMacAddress()));
   } else {
     //The Connections
     if (device_id[0]) {
-      JsonArray connections = device.createNestedArray("connections");
+      JsonArray connections = device.createNestedArray("cns");
       JsonArray connection_mac = connections.createNestedArray();
       connection_mac.add("mac");
       connection_mac.add(device_id);
@@ -374,11 +382,11 @@ void createDiscovery(const char* sensor_type,
     }
 
     if (device_manufacturer[0]) {
-      device["manufacturer"] = device_manufacturer;
+      device["mf"] = device_manufacturer;
     }
 
     if (device_model[0]) {
-      device["model"] = device_model;
+      device["mdl"] = device_model;
     }
 
     // generate unique device name by adding the second half of the device_id only if device_name and device_id are different

main/main.ino

@@ -78,6 +78,7 @@ bool ready_to_sleep = false;
 
 StaticJsonDocument<JSON_MSG_BUFFER> modulesBuffer;
 JsonArray modules = modulesBuffer.to<JsonArray>();
+bool ethConnected = false;
 
 #ifndef ZgatewayGFSunInverter
 // Arduino IDE compiles, it automatically creates all the header declarations for all the functions you have in your *.ino file.
@@ -264,7 +265,6 @@ bool ProcessLock = false; // Process lock when we want to use a critical functio
 #  ifdef ESP32_ETHERNET
 #    include <ETH.h>
 void WiFiEvent(WiFiEvent_t event);
-static bool esp32EthConnected = false;
 #  endif
 
 #  include <WiFiClientSecure.h>
@@ -952,18 +952,27 @@ void setup() {
   modules.add(ZactuatorONOFF);
 #  endif
 
-#  ifdef ESP32_ETHERNET
+#  if defined(ESPWifiManualSetup)
-  setup_ethernet_esp32();
-#  else // WIFI ESP
-#    if defined(ESPWifiManualSetup)
   setup_wifi();
-#    else
+#  else
-  // In failSafeMode we don't want to setup wifi manager as it has already been done before
+  if (loadConfigFromFlash()) {
-  if (!failSafeMode) setup_wifimanager(false);
+    Log.notice(F("Config loaded from flash" CR));
+#    ifdef ESP32_ETHERNET
+    setup_ethernet_esp32();
 #    endif
+    if (!failSafeMode && !ethConnected) setup_wifimanager(false);
+  } else {
+#    ifdef ESP32_ETHERNET
+    setup_ethernet_esp32();
+#    endif
+    Log.notice(F("No config in flash, launching wifi manager" CR));
+    // In failSafeMode we don't want to setup wifi manager as it has already been done before
+    if (!failSafeMode) setup_wifimanager(false);
+  }
+
+#  endif
   Log.trace(F("OpenMQTTGateway mac: %s" CR), WiFi.macAddress().c_str());
   Log.trace(F("OpenMQTTGateway ip: %s" CR), WiFi.localIP().toString().c_str());
-#  endif
 
   setOTA();
 #else // In case of arduino platform
@@ -1189,9 +1198,9 @@ void setup() {
 bool wifi_reconnect_bypass() {
   uint8_t wifi_autoreconnect_cnt = 0;
 #  ifdef ESP32
-  while (WiFi.status() != WL_CONNECTED && wifi_autoreconnect_cnt < maxConnectionRetryWifi) {
+  while (WiFi.status() != WL_CONNECTED && wifi_autoreconnect_cnt < maxConnectionRetryNetwork) {
 #  else
-  while (WiFi.waitForConnectResult() != WL_CONNECTED && wifi_autoreconnect_cnt < maxConnectionRetryWifi) {
+  while (WiFi.waitForConnectResult() != WL_CONNECTED && wifi_autoreconnect_cnt < maxConnectionRetryNetwork) {
 #  endif
     Log.notice(F("Attempting Wifi connection with saved AP: %d" CR), wifi_autoreconnect_cnt);
 
@@ -1202,7 +1211,7 @@ bool wifi_reconnect_bypass() {
     delay(1000);
     wifi_autoreconnect_cnt++;
   }
-  if (wifi_autoreconnect_cnt < maxConnectionRetryWifi) {
+  if (wifi_autoreconnect_cnt < maxConnectionRetryNetwork) {
     return true;
   } else {
     return false;
@@ -1316,6 +1325,10 @@ void setupTLS(bool self_signed, uint8_t index) {
 #  endif
   }
 }
+#else
+bool wifi_reconnect_bypass() {
+  return true;
+}
 #endif
 
 /*
@@ -1372,7 +1385,7 @@ void setup_wifi() {
     failure_number_ntwk++;
 #  if defined(ESP32) && defined(ZgatewayBT)
     if (lowpowermode) {
-      if (failure_number_ntwk > maxConnectionRetryWifi) {
+      if (failure_number_ntwk > maxConnectionRetryNetwork) {
         lowPowerESP32();
       }
     } else {
@@ -1494,28 +1507,9 @@ void saveMqttConfig() {
   configFile.close();
 }
 
-void setup_wifimanager(bool reset_settings) {
+bool loadConfigFromFlash() {
-  delay(10);
-  WiFi.mode(WIFI_STA);
-
-  if (reset_settings)
-    eraseAndRestart();
-
-  String s = WiFi.macAddress();
-#  ifdef USE_MAC_AS_GATEWAY_NAME
-  sprintf(gateway_name, "%.2s%.2s%.2s%.2s%.2s%.2s",
-          s.c_str(), s.c_str() + 3, s.c_str() + 6, s.c_str() + 9, s.c_str() + 12, s.c_str() + 15);
-  snprintf(WifiManager_ssid, MAC_NAME_MAX_LEN, "%s_%.2s%.2s", Gateway_Short_Name, s.c_str(), s.c_str() + 3);
-  strcpy(ota_hostname, WifiManager_ssid);
-  Log.notice(F("OTA Hostname: %s.local" CR), ota_hostname);
-#  endif
-#  ifdef WM_PWD_FROM_MAC // From ESP Mac Address, last 8 digits as the password
-  sprintf(WifiManager_password, "%.2s%.2s%.2s%.2s",
-          s.c_str() + 6, s.c_str() + 9, s.c_str() + 12, s.c_str() + 15);
-#  endif
-
-  //read configuration from FS json
   Log.trace(F("mounting FS..." CR));
+  bool result = false;
 
   if (SPIFFS.begin()) {
     Log.trace(F("mounted file system" CR));
@@ -1561,12 +1555,35 @@ void setup_wifimanager(bool reset_settings) {
           strcpy(ota_pass, json["ota_pass"]);
         if (json.containsKey("ota_server_cert"))
           ota_server_cert = json["ota_server_cert"].as<const char*>();
+        result = true;
       } else {
         Log.warning(F("failed to load json config" CR));
       }
       configFile.close();
     }
   }
+  return result;
+}
+
+void setup_wifimanager(bool reset_settings) {
+  delay(10);
+  WiFi.mode(WIFI_STA);
+
+  if (reset_settings)
+    eraseAndRestart();
+
+  String s = WiFi.macAddress();
+#  ifdef USE_MAC_AS_GATEWAY_NAME
+  sprintf(gateway_name, "%.2s%.2s%.2s%.2s%.2s%.2s",
+          s.c_str(), s.c_str() + 3, s.c_str() + 6, s.c_str() + 9, s.c_str() + 12, s.c_str() + 15);
+  snprintf(WifiManager_ssid, MAC_NAME_MAX_LEN, "%s_%.2s%.2s", Gateway_Short_Name, s.c_str(), s.c_str() + 3);
+  strcpy(ota_hostname, WifiManager_ssid);
+  Log.notice(F("OTA Hostname: %s.local" CR), ota_hostname);
+#  endif
+#  ifdef WM_PWD_FROM_MAC // From ESP Mac Address, last 8 digits as the password
+  sprintf(WifiManager_password, "%.2s%.2s%.2s%.2s",
+          s.c_str() + 6, s.c_str() + 9, s.c_str() + 12, s.c_str() + 15);
+#  endif
 
   wifiManager.setDebugOutput(WM_DEBUG_LEVEL);
 
@@ -1623,6 +1640,10 @@ void setup_wifimanager(bool reset_settings) {
   //set minimum quality of signal so it ignores AP's under that quality
   wifiManager.setMinimumSignalQuality(MinimumWifiSignalQuality);
 
+#  ifdef ESP32_ETHERNET
+  wifiManager.setBreakAfterConfig(true); // If ethernet is used, we don't want to block the connection by keeping the portal up
+#  endif
+
   if (!wifi_reconnect_bypass()) // if we didn't connect with saved credential we start Wifimanager web portal
   {
 #  ifdef ESP32
@@ -1643,21 +1664,23 @@ void setup_wifimanager(bool reset_settings) {
     //if it does not connect it starts an access point with the specified name
     //and goes into a blocking loop awaiting configuration
     if (!wifiManager.autoConnect(WifiManager_ssid, WifiManager_password)) {
-      Log.warning(F("failed to connect and hit timeout" CR));
+      if (!ethConnected) { // If we are using ethernet, we don't want to restart the ESP
-      delay(3000);
+        Log.warning(F("failed to connect and hit timeout" CR));
+        delay(3000);
 
 #  ifdef ESP32
-      /* Workaround for bug in arduino core that causes the AP to become unsecure on reboot */
+        /* Workaround for bug in arduino core that causes the AP to become unsecure on reboot */
-      esp_wifi_set_mode(WIFI_MODE_AP);
+        esp_wifi_set_mode(WIFI_MODE_AP);
-      esp_wifi_start();
+        esp_wifi_start();
-      wifi_config_t conf;
+        wifi_config_t conf;
-      esp_wifi_get_config(WIFI_IF_AP, &conf);
+        esp_wifi_get_config(WIFI_IF_AP, &conf);
-      conf.ap.ssid_hidden = 1;
+        conf.ap.ssid_hidden = 1;
-      esp_wifi_set_config(WIFI_IF_AP, &conf);
+        esp_wifi_set_config(WIFI_IF_AP, &conf);
 #  endif
 
-      //restart and try again
+        //restart and try again
-      ESPRestart(3);
+        ESPRestart(3);
+      }
     }
     InfoIndicatorOFF();
     ErrorIndicatorOFF();
@@ -1721,9 +1744,10 @@ void setup_ethernet_esp32() {
   ethBeginSuccess = ETH.begin();
 #    endif
   Log.trace(F("Connecting to Ethernet" CR));
-  while (!esp32EthConnected) {
+  while (!ethConnected && failure_number_ntwk <= maxConnectionRetryNetwork) {
     delay(500);
     Log.trace(F("." CR));
+    failure_number_ntwk++;
   }
 }
 
@@ -1740,15 +1764,15 @@ void WiFiEvent(WiFiEvent_t event) {
       Log.trace(F("OpenMQTTGateway MAC: %s" CR), ETH.macAddress().c_str());
       Log.trace(F("OpenMQTTGateway IP: %s" CR), ETH.localIP().toString().c_str());
       Log.trace(F("OpenMQTTGateway link speed: %d Mbps" CR), ETH.linkSpeed());
-      esp32EthConnected = true;
+      ethConnected = true;
       break;
     case ARDUINO_EVENT_ETH_DISCONNECTED:
       Log.error(F("Ethernet Disconnected" CR));
-      esp32EthConnected = false;
+      ethConnected = false;
       break;
     case ARDUINO_EVENT_ETH_STOP:
       Log.error(F("Ethernet Stopped" CR));
-      esp32EthConnected = false;
+      ethConnected = false;
       break;
     default:
       break;
@@ -1824,11 +1848,10 @@ void loop() {
   }
 
 #if defined(ESP8266) || defined(ESP32)
-#  ifdef ESP32_ETHERNET
+  if (ethConnected || WiFi.status() == WL_CONNECTED) {
-  if (esp32EthConnected) {
+    if (ethConnected && WiFi.status() == WL_CONNECTED) {
-#  else
+      WiFi.disconnect(); // we disconnect the wifi as we are connected to ethernet
-  if (WiFi.status() == WL_CONNECTED) {
+    }
-#  endif
     ArduinoOTA.handle();
 #else
   if ((Ethernet.hardwareStatus() != EthernetW5100 && Ethernet.linkStatus() == LinkON) || (Ethernet.hardwareStatus() == EthernetW5100)) { //we are able to detect disconnection only on w5200 and w5500
@@ -2015,7 +2038,7 @@ void loop() {
     esp_task_wdt_reset();
 #endif
     connected = false;
-    Log.warning(F("Network disconnected:" CR));
+    Log.warning(F("Network disconnected" CR));
     ErrorIndicatorON();
     delay(2000); // add a delay to avoid ESP32 crash and reset
 #ifdef ESP32
@@ -2023,14 +2046,7 @@ void loop() {
 #endif
     ErrorIndicatorOFF();
     delay(2000);
-#if defined(ESP8266) || defined(ESP32) && !defined(ESP32_ETHERNET)
+    wifi_reconnect_bypass();
-#  ifdef ESP32 // If used with ESP8266 this method prevent the reconnection
-    WiFi.reconnect();
-#  endif
-    Log.warning(F("wifi" CR));
-#else
-    Log.warning(F("ethernet" CR));
-#endif
   }
 // Function that doesn't need an active connection
 #if defined(ZboardM5STICKC) || defined(ZboardM5STICKCP) || defined(ZboardM5STACK) || defined(ZboardM5TOUGH)
@@ -2176,22 +2192,23 @@ String stateMeasures() {
 #      endif
   SYSdata["freestack"] = uxTaskGetStackHighWaterMark(NULL);
 #    endif
+
+  SYSdata["ethernet"] = ethConnected;
+  if (ethConnected) {
 #    ifdef ESP32_ETHERNET
-  SYSdata["mac"] = (char*)ETH.macAddress().c_str();
+    SYSdata["mac"] = (char*)ETH.macAddress().c_str();
-  SYSdata["ip"] = ip2CharArray(ETH.localIP());
+    SYSdata["ip"] = ip2CharArray(ETH.localIP());
-  ETH.fullDuplex() ? SYSdata["fd"] = (bool)"true" : SYSdata["fd"] = (bool)"false";
+    ETH.fullDuplex() ? SYSdata["fd"] = (bool)"true" : SYSdata["fd"] = (bool)"false";
-  SYSdata["linkspeed"] = (int)ETH.linkSpeed();
+    SYSdata["linkspeed"] = (int)ETH.linkSpeed();
-#    else
-  long rssi = WiFi.RSSI();
-  SYSdata["rssi"] = rssi;
-  String SSID = WiFi.SSID();
-  SYSdata["SSID"] = SSID;
-  String BSSID = WiFi.BSSIDstr();
-  SYSdata["BSSID"] = BSSID;
-  SYSdata["ip"] = ip2CharArray(WiFi.localIP());
-  String mac = WiFi.macAddress();
-  SYSdata["mac"] = (char*)mac.c_str();
 #    endif
+  } else {
+    SYSdata["rssi"] = (long)WiFi.RSSI();
+    SYSdata["SSID"] = (char*)WiFi.SSID().c_str();
+    SYSdata["BSSID"] = (char*)WiFi.BSSIDstr().c_str();
+    SYSdata["ip"] = ip2CharArray(WiFi.localIP());
+    SYSdata["mac"] = (char*)WiFi.macAddress().c_str();
+  }
+
 #  endif
 #  ifdef ZgatewayBT
 #    ifdef ESP32

platformio.ini

@@ -99,7 +99,6 @@ extra_configs =
 ;default_envs = esp32c3-dev-c2-ble
 ;default_envs = lolin_c3_mini
 ;default_envs = thingpulse-espgateway
-;default_envs = thingpulse-espgateway-ethernet
 
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 ;                              ENVIRONMENTS PARAMETERS                                 ;