[BT] Device tracker presence detection (#1548)

Add device tracker detection and automatic creation following HA discovery convention
So as to trigger the away state, add an offline status to the identified BLE trackers

docs/use/ble.md

@@ -29,6 +29,26 @@ Once the data has been transmitted to the MQTT broker, it can be easily integrat
 
 Examples of compatible sensors among [our list](https://decoder.theengs.io/devices/devices_by_brand.html: Mi Flora, Mi jia, LYWDS02, LYWSD03MMC, ClearGrass, Mi scale, iBBQ, TPMS
 
+## Receiving signals from BLE tracker devices for Presence detection
+By default the gateway will detect the BLE trackers from Tile, NUT, TAGIT, ITAG, MiBand, Amazfit and RuuviTag and create automaticaly a device tracker entity following the Home Assistant discovery convention (if the auto discovery is activated).
+The entity can be attached to a person to leverage presence detection. The `away` or `not home` state is triggered if the BLE tracker is not detected during the timer defined by `presenceawaytimer`.
+
+![](../img/OpenMQTTGateway-BLE-tracker-Home-Assistant.png)
+
+If you have multiple gateways, your BLE trackers may not be detected temporary by one gateway but still by the others. In this case you will see the tracker appears offline briefly and online again once it is detected by the others gateways.
+
+By default `presenceawaytimer` is set to 120s, you can change it with the following command (ms)
+
+`mosquitto_pub -t home/OpenMQTTGateway/commands/MQTTtoBT/config -m '{"presenceawaytimer":66000}'`
+
+Generally BLE devices will not broadcast if they are paired so you may need to ensure your beacons is unpaired/disconnected before it will be seen by the gateway.
+
+Consider the distance estimation as a beta feature.
+
+Note that you can find apps to simulate beacons and do some tests like [Beacon simulator](https://play.google.com/store/apps/details?id=net.alea.beaconsimulator)
+
+iOS version >=10 devices advertise without an extra app MAC address, nevertheless this address [changes randomly](https://github.com/1technophile/OpenMQTTGateway/issues/71) and cannot be used for presence detection. You must install an app to advertise a fixed MAC address.
+
 ## Setting a white or black list
 A black list is a list of MAC addresses that will never be published by OMG
 to set black list
@@ -158,33 +178,6 @@ If you want to change this characteristic:
 With Home Assistant, this command is directly available through MQTT auto discovery as a switch into the HASS OpenMQTTGateway device entities list.
 :::
 
-## Receiving signals from BLE beacon devices for Presence detection
-
-Subscribe to all the messages with mosquitto or open your MQTT client software:
-
-`    sudo mosquitto_sub -t +/# -v`
-
-_NOTE: HM-10 or HM-11 module needed if you are not using ESP32; configure in `User_config.h`_
-
-The BT gateway module for OpenMQTTGateway enables the detection of BLE beacons and their signal strength.  Generally BLE devices will not broadcast if they are paired so you may need to ensure your beacons is unpaired before it will be seen by the gateway.
-
-If beacons are detected the gateway will periodically publish messages to MQTT (beacons must not be paired, see above):
-
-```
-home/OpenMQTTGateway/BTtoMQTT/45E174126E00 {"id":"45:e1:74:12:6e:00","rssi":-89,"distance":21.51847,"servicedata":"fe0000000000000000000000000000000000000000"}
-```
-```
-home/OpenMQTTGateway/BTtoMQTT/C7FaaD132C00 {"id":"c7:fa:ad:13:2c:00","rssi":-68,"distance":2.799256,"servicedata":"drfgdrgdsrgesrdgdrgdregesrgtrhtyhtfyhdtyhh"}
-```
-
-The subtopic after `home/BTtoMQTT/` is the MAC address of the Bluetooth low energy beacon.  The rssi value is the [RSSI signal level](https://www.metageek.com/training/resources/understanding-rssi.html) from which you may deduce the relative distance to the device.
-Consider the distance as a beta feature as currently we are not retrieving the emitting power of the beacon to make it more accurate.
-
-Note that you can find apps to simulate beacons and do some tests like [Beacon simulator](https://play.google.com/store/apps/details?id=net.alea.beaconsimulator)
-
-iOS version >=10 devices advertise without an extra app MAC address, nevertheless this address [changes randomly](https://github.com/1technophile/OpenMQTTGateway/issues/71) and cannot be used for presence detection. You must install an app to advertise a fixed MAC address.
-
-
 ## Setting if the gateway publish into Home Assistant Home presence topic
 
 If you want to publish to Home Assistant presence topic, you can activate this function by the HASS interface (this command is auto discovered), [here is a yaml example](../integrate/home_assistant.md#mqtt-room-presence).

main/ZgatewayBT.ino

@@ -98,6 +98,7 @@ void BTConfig_init() {
   BTConfig.pubAdvData = pubBLEAdvData;
   BTConfig.pubBeaconUuidForTopic = useBeaconUuidForTopic;
   BTConfig.ignoreWBlist = false;
+  BTConfig.presenceAwayTimer = PresenceAwayTimer;
 }
 
 unsigned long timeBetweenConnect = 0;
@@ -124,6 +125,7 @@ void stateBTMeasures(bool start) {
   jo["pubadvdata"] = BTConfig.pubAdvData;
   jo["pubBeaconUuidForTopic"] = BTConfig.pubBeaconUuidForTopic;
   jo["ignoreWBlist"] = BTConfig.ignoreWBlist;
+  jo["presenceawaytimer"] = BTConfig.presenceAwayTimer;
   jo["btqblck"] = btQueueBlocked;
   jo["btqsum"] = btQueueLengthSum;
   jo["btqsnd"] = btQueueLengthCount;
@@ -143,6 +145,16 @@ void BTConfig_fromJson(JsonObject& BTdata, bool startup = false) {
   Config_update(BTdata, "bleconnect", BTConfig.bleConnect);
   // Identify AdaptiveScan deactivation to pass to continuous mode or activation to come back to default settings
   if (startup == false) {
+    if (BTdata.containsKey("hasspresence") && BTdata["hasspresence"] == false && BTConfig.presenceEnable == true) {
+      BTdata["adaptivescan"] = true;
+#  ifdef ZmqttDiscovery
+      // Remove discovered entities
+      eraseTopic("number", (char*)getUniqueId("presenceawaytimer", "").c_str());
+#  endif
+    } else if (BTdata.containsKey("hasspresence") && BTdata["hasspresence"] == true && BTConfig.presenceEnable == false) {
+      BTdata["adaptivescan"] = false;
+    }
+
     if (BTdata.containsKey("adaptivescan") && BTdata["adaptivescan"] == false && BTConfig.adaptiveScan == true) {
       BTdata["interval"] = MinTimeBtwScan;
       BTdata["intervalacts"] = MinTimeBtwScan;
@@ -159,9 +171,12 @@ void BTConfig_fromJson(JsonObject& BTdata, bool startup = false) {
     }
   }
   Config_update(BTdata, "adaptivescan", BTConfig.adaptiveScan);
+  // Home Assistant presence message
+  Config_update(BTdata, "hasspresence", BTConfig.presenceEnable);
 #  ifdef ZmqttDiscovery
   // Create discovery entities
   btScanParametersDiscovery();
+  btPresenceParametersDiscovery();
 #  endif
   // Time before before active scan
   // Scan interval set
@@ -177,12 +192,12 @@ void BTConfig_fromJson(JsonObject& BTdata, bool startup = false) {
   Config_update(BTdata, "onlysensors", BTConfig.pubOnlySensors);
   // publish devices which randomly change their MAC addresses
   Config_update(BTdata, "randommacs", BTConfig.pubRandomMACs);
-  // Home Assistant presence message
-  Config_update(BTdata, "hasspresence", BTConfig.presenceEnable);
   // Home Assistant presence message topic
   Config_update(BTdata, "presenceTopic", BTConfig.presenceTopic);
   // Home Assistant presence message use iBeacon UUID
   Config_update(BTdata, "presenceUseBeaconUuid", BTConfig.presenceUseBeaconUuid);
+  // Timer to trigger a device state as offline if not seen
+  Config_update(BTdata, "presenceawaytimer", BTConfig.presenceAwayTimer);
   // MinRSSI set
   Config_update(BTdata, "minrssi", BTConfig.minRssi);
   // Send undecoded device data
@@ -230,6 +245,7 @@ void BTConfig_fromJson(JsonObject& BTdata, bool startup = false) {
     jo["pubadvdata"] = BTConfig.pubAdvData;
     jo["pubBeaconUuidForTopic"] = BTConfig.pubBeaconUuidForTopic;
     jo["ignoreWBlist"] = BTConfig.ignoreWBlist;
+    jo["presenceawaytimer"] = BTConfig.presenceAwayTimer;
     // Save config into NVS (non-volatile storage)
     String conf = "";
     serializeJson(jsonBuffer, conf);
@@ -408,11 +424,12 @@ void createOrUpdateDevice(const char* mac, uint8_t flags, int model, int mac_typ
     device->connect = flags & device_flags_connect;
     device->macType = mac_type;
     device->sensorModel_id = model;
+    device->lastUpdate = millis();
     devices.push_back(device);
     newDevices++;
   } else {
     Log.trace(F("update %s" CR), mac);
-
+    device->lastUpdate = millis();
     device->macType = mac_type;
 
     if (flags & device_flags_isDisc) {
@@ -439,6 +456,30 @@ void createOrUpdateDevice(const char* mac, uint8_t flags, int model, int mac_typ
   xSemaphoreGive(semaphoreCreateOrUpdateDevice);
 }
 
+void updateDevicesStatus() {
+  for (vector<BLEdevice*>::iterator it = devices.begin(); it != devices.end(); ++it) {
+    BLEdevice* p = *it;
+    unsigned long now = millis();
+    if (p->sensorModel_id == TheengsDecoder::BLE_ID_NUM::NUT ||
+        p->sensorModel_id == TheengsDecoder::BLE_ID_NUM::MIBAND ||
+        p->sensorModel_id == TheengsDecoder::BLE_ID_NUM::TAGIT ||
+        p->sensorModel_id == TheengsDecoder::BLE_ID_NUM::TILE ||
+        p->sensorModel_id == TheengsDecoder::BLE_ID_NUM::ITAG ||
+        p->sensorModel_id == TheengsDecoder::BLE_ID_NUM::RUUVITAG_RAWV1 ||
+        p->sensorModel_id == TheengsDecoder::BLE_ID_NUM::RUUVITAG_RAWV2) { // We apply the offline status only for tracking device, can be extended further to all the devices
+      if ((p->lastUpdate != 0) && (p->lastUpdate < (now - BTConfig.presenceAwayTimer) && (now > BTConfig.presenceAwayTimer)) &&
+          (BTConfig.ignoreWBlist || ((!oneWhite || isWhite(p)) && !isBlack(p)))) { // Only if WBlist is disabled OR ((no white MAC OR this MAC is white) AND not a black listed MAC)) {
+        JsonObject BLEdata = getBTJsonObject();
+        BLEdata["id"] = p->macAdr;
+        BLEdata["state"] = "offline";
+        pubBT(BLEdata);
+        // We set the lastUpdate to 0 to avoid replublishing the offline state
+        p->lastUpdate = 0;
+      }
+    }
+  }
+}
+
 void dumpDevices() {
   for (vector<BLEdevice*>::iterator it = devices.begin(); it != devices.end(); ++it) {
     BLEdevice* p = *it;
@@ -449,6 +490,7 @@ void dumpDevices() {
     Log.trace(F("isBlkL %d" CR), p->isBlkL);
     Log.trace(F("connect %d" CR), p->connect);
     Log.trace(F("sensorModel_id %d" CR), p->sensorModel_id);
+    Log.trace(F("LastUpdate %u" CR), p->lastUpdate);
   }
 }
 
@@ -646,6 +688,8 @@ void procBLETask(void* pvParameters) {
       } else {
         Log.trace(F("Filtered MAC device" CR));
       }
+      if (BTConfig.presenceEnable)
+        updateDevicesStatus();
     }
     delete (advertisedDevice);
   }
@@ -869,6 +913,7 @@ void setupBT() {
   Log.notice(F("Active BLE scan interval: %d" CR), BTConfig.intervalActiveScan);
   Log.notice(F("minrssi: %d" CR), -abs(BTConfig.minRssi));
   Log.notice(F("Low Power Mode: %d" CR), lowpowermode);
+  Log.notice(F("Presence Away Timer: %d" CR), BTConfig.presenceAwayTimer);
 
   atomic_init(&forceBTScan, 0); // in theory, we don't need this
   atomic_init(&jsonBTBufferQueueNext, 0); // in theory, we don't need this
@@ -983,6 +1028,21 @@ void launchBTDiscovery(bool overrideDiscovery) {
                               model.c_str(), brand.c_str(), model_id.c_str(), macWOdots.c_str(), false,
                               stateClassMeasurement);
             }
+            if (p->sensorModel_id == TheengsDecoder::BLE_ID_NUM::NUT ||
+                p->sensorModel_id == TheengsDecoder::BLE_ID_NUM::MIBAND ||
+                p->sensorModel_id == TheengsDecoder::BLE_ID_NUM::TAGIT ||
+                p->sensorModel_id == TheengsDecoder::BLE_ID_NUM::TILE ||
+                p->sensorModel_id == TheengsDecoder::BLE_ID_NUM::ITAG ||
+                p->sensorModel_id == TheengsDecoder::BLE_ID_NUM::RUUVITAG_RAWV1 ||
+                p->sensorModel_id == TheengsDecoder::BLE_ID_NUM::RUUVITAG_RAWV2) {
+              createDiscovery("device_tracker",
+                              discovery_topic.c_str(), entity_name.c_str(), unique_id.c_str(),
+                              will_Topic, "occupancy", "{% if value_json.rssi -%}home{%- else -%}not_home{%- endif %}",
+                              "", "", "",
+                              0, "", "", false, "",
+                              model.c_str(), brand.c_str(), model_id.c_str(), macWOdots.c_str(), false,
+                              stateClassNone);
+            }
           }
         }
       } else {

main/ZmqttDiscovery.ino

@@ -295,6 +295,8 @@ void createDiscovery(const char* sensor_type,
   }
   if (payload_off[0])
     sensor["pl_off"] = payload_off; //payload_off
+  if (strcmp(sensor_type, "device_tracker") == 0)
+    sensor["source_type "] = "bluetooth_le"; // payload_install for update
   if (off_delay != 0)
     sensor["off_delay"] = off_delay; //off_delay
   if (payload_available[0])
@@ -386,8 +388,22 @@ void eraseTopic(const char* sensor_type, const char* unique_id) {
 }
 
 #  ifdef ZgatewayBT
+void btPresenceParametersDiscovery() {
+  if (BTConfig.presenceEnable) {
+    createDiscovery("number", //set Type
+                    subjectBTtoMQTT, "BT: Presence detection timer", (char*)getUniqueId("presenceawaytimer", "").c_str(), //set state_topic,name,uniqueId
+                    will_Topic, "", "{{ value_json.presenceawaytimer/60000 }}", //set availability_topic,device_class,value_template,
+                    "{\"presenceawaytimer\":{{value*60000}},\"save\":true}", "", "min", //set,payload_on,payload_off,unit_of_meas,
+                    0, //set  off_delay
+                    Gateway_AnnouncementMsg, will_Message, true, subjectMQTTtoBTset, //set,payload_available,payload_not available   ,is a gateway entity, command topic
+                    "", "", "", "", false, // device name, device manufacturer, device model, device ID, retain,
+                    stateClassNone //State Class
+    );
+  }
+}
+
 void btScanParametersDiscovery() {
-  if (BTConfig.adaptiveScan == false) {
+  if (!BTConfig.adaptiveScan) {
     createDiscovery("number", //set Type
                     subjectBTtoMQTT, "BT: Interval between scans", (char*)getUniqueId("interval", "").c_str(), //set state_topic,name,uniqueId
                     will_Topic, "", "{{ value_json.interval/1000 }}", //set availability_topic,device_class,value_template,
@@ -1105,6 +1121,8 @@ void pubMqttDiscovery() {
 
   btScanParametersDiscovery();
 
+  btPresenceParametersDiscovery();
+
   createDiscovery("switch", //set Type
                   subjectBTtoMQTT, "BT: Publish HASS presence", (char*)getUniqueId("hasspresence", "").c_str(), //set state_topic,name,uniqueId
                   will_Topic, "", "{{ value_json.hasspresence }}", //set availability_topic,device_class,value_template,

main/config_BT.h

@@ -89,6 +89,9 @@ extern int btQueueLengthCount;
 #ifndef TimeBtwConnect
 #  define TimeBtwConnect 3600000 //define default time between BLE connection attempt (not used for immediate actions); in milliseconds
 #endif
+#ifndef PresenceAwayTimer
+#  define PresenceAwayTimer 120000 //define the time between Offline Status update for the sensors
+#endif
 
 #ifndef BLEScanDuplicateCacheSize
 #  define BLEScanDuplicateCacheSize 200
@@ -160,6 +163,7 @@ struct BTConfig_s {
   bool pubAdvData; // Publish advertisement data
   bool pubBeaconUuidForTopic; // Use iBeacon UUID as topic, instead of sender (random) MAC address
   bool ignoreWBlist; // Disable Whitelist & Blacklist
+  unsigned long presenceAwayTimer; //Timer that trigger a tracker state as offline if not seen
 };
 
 // Global struct to store live BT configuration data
@@ -209,6 +213,7 @@ struct BLEdevice {
   bool isBlkL;
   bool connect;
   int sensorModel_id;
+  unsigned long lastUpdate;
 };
 
 class BLEconectable {