mqtt discovery, compare & restore

[robmwalsh]

Is your feature request related to a problem? Please describe.
I briefly thought I'd accidentally reset ALL of my devices - turns out the issue was that my router needed a restart phew, but this made me think it'd be a good time to backup all of my device configs, which is how (after being disappointed to learn that the config export is binary) I found this project. I'm also a big fan of version control which seems to be very lacking in the home automation space in general, but it looks like most of the parts are already here!

Describe the solution you'd like

1. autodiscovery and backup of all tasmota devices using specified MQTT server/credentials
2. save all decoded configs in specified folder, ideally with option for mac address for filename, overwrite if exists
3. option to restore all from a folder, using mqtt server to get IPs for mac addresses, ideally with a check to see if config has changed to avoid writing unchanged configs
4. profit

Describe alternatives you've considered
none

Additional context
intention for this is making this work with git, allowing you to commit changes when you do something useful, checkout/roll back to a known good state when you're experimenting and break your house (or are part way though doing something but have to stop and would like everything to keep working until you get back to working on it)

[curzon01]

This is how I do it: A bash script runs at night that scans all devices and creates a json backup with decode-config; then git runs over the directory. I can make the Linux script available as as.si.

I put that in "Discussion" because I don't see such a solution as part of decode-config itself.

[robmwalsh]

Thanks @curzon01, if you can make the script available that'd be much appreciated. Perhaps a gist?

[curzon01]

it does not what you described (that's a nice idea but dev too time consuming I currently do not have) - it simply scans IPs within my network range and brute force to run decode-config on it; as simple as possible

[robmwalsh]

Right, that's another option I suppose! I've investigated a few options and think that I'll create a HA add-on which would be built around decode-config. I think that'll create the biggest benefit for the community as it'll be the easiest way for people to version control their tasmota configs. Perhaps in future it could incorporate other stuff like HA config, but that's a problem for another time.

[pgorod]

@robmwalsh did you get anything going?

@curzon01 did you ever publish your script "as-is"? If so, can you please point me there. Thanks!

[curzon01]

Here are my very simple bash backup script saving json and dmp file in one call in two loops:

• first loop saves tasmotas by topics config via MQTT
• second loop saves tasmotas by IP via HTTP
• finally git repo will be updated

Script uses nmap and awk for host detection and decode-config.py

For better protection decode-config.py is called using a separate config file tasmota.conf containing all passwords

tasmota-backups.zip

[pgorod]

Thanks! Can I ask, what is the reason to use the first loop for MQTT? I am just starting with MQTT so I am not very familiar. Is it also to backup configurations? Or is it meant to save the history of events etc?

[curzon01]

Access via MQTT saves exactly the same data as a backup via HTTP or directly on the web interface "Configuration -> Backup Configuration".
Tasmota always delivers and accepts the same binary data of the complete configuration (see overview picture left side "Source").
There is no difference whether you request or write back the binary configuration data via web interface, HTTP or MQTT - it is always the entire configuration flash data. decode-config is your converter between left (Source) and right (Output) side.

The disadvantage of IP-based access of Tasmota (HTTP) is that you can only access devices that you can reach in your own network.

The MQTT backup first loop saves all my remote devices to which I only have access via MQTT and not by IP - they are somewhere in the world, but not in my network and I can't connect via IP.

[pgorod]

Thanks for the detailed explanation, I appreciate it. 👍

FYI I started writing a script but didn't finish it (my project had to change a bit and for now we can't use Tasmota, maybe later). It doesn't do much but it does include a nice advantage compared to yours - it uses GNU parallel to do all the work concurrently, making things quite fast. The proof of concept is working regarding the parallel execution. I'll try to remember coming back here and sharing it if I ever finish it.

[curzon01]

I wonder why a framework is needed, just start the processes in bash using & in parallel too:

#!/bin/bash
BACKUPTOOL=/usr/local/bin/decode-config.py
BACKUPCONF=/usr/local/etc/tasmota.conf
BACKUPDIR="/srv/data/backup"

# create json and dmp file directories
mkdir -p "${BACKUPDIR}/json"
mkdir -p "${BACKUPDIR}/dmp"

backup_by_mqtt() {
    ${BACKUPTOOL} -c ${BACKUPCONF} --fulltopic tele/tasmota/$1 -o "${BACKUPDIR}/json/@F.json" -o "${BACKUPDIR}/dmp/@F.dmp"
}

backup_by_ip() {
    ip=$1
    # get hostname from IP
    HOSTNAME=$(host $ip | awk '/domain name pointer/ { print $5 }')
    HOST=${HOSTNAME%%.*}
    # check if device has an open port 80
    nmap -p 80 $ip|grep "open" > /dev/null
    if [ "$?" -eq 0 ]
    then
        echo "Backup by host $HOST [$ip]"
        FILENAME="Config_${HOST}_@d"
        ${BACKUPTOOL} -c ${BACKUPCONF} -s $ip -o "${BACKUPDIR}/json/${FILENAME}.json" -o "${BACKUPDIR}/dmp/${FILENAME}.dmp"
    else
        echo "$HOST [$ip] OFFLINE"
    fi
}

echo "========================================================================"
# MQTT devices
echo "Backup MQTT based devices"
for topic in tasmota_0EBC55 tasmota_FCED49 tasmota_98CC13 tasmota_A75BBC
do
    echo "Backup by MQTT topic tele/tasmota/$topic"
    backup_by_mqtt "$topic" &
done

echo "Backup IP based devices"
# IP range for Tasmota devices 172.17.12.80 - 172.17.12.169
for i in $(seq 80 169)
do
    backup_by_ip "172.17.12.$i" &
done
echo "========================================================================"

echo "Waiting for jobs finished..."
while [ "$(jobs -r -p)" != "" ]; do
    sleep 0.5
done

# commit changes in json files only
STAMP=$(date '+%Y-%m-%d %H:%M:%S')
git -C "${BACKUPDIR}" add json
git -C "${BACKUPDIR}" commit -a -m "daily autocommit ${STAMP}"

exit 0

[pgorod]

I like your solution. I'd used & for single commands but I've never used it "massively" like this. I also learned new tricks with this part of your script:

echo "Waiting for jobs finished..."
while [ "$(jobs -r -p)" != "" ]; do
    sleep 0.5
done

Nice! 😎

My unfinished script looks like this:

#!/bin/bash
set -e

single() {
    ping -q -c 3 ${1} > /dev/null
    pingResponse=$?

    if [ $pingResponse -eq 0 ];then
       curlResponse=`curl --connect-timeout 10 ${1}/cm?cmnd=Status0 | jq  '.StatusNET | .Mac + " " + .Hostname'`
       echo ${1} $curlResponse >> out.txt
    fi
}

export -f single
rm out.txt
parallel -j0 --tag single 10.0.0.{1} ::: {1..254}
sort -t. -n +3.0 out.txt -o out.txt

It scans all the addresses in the subnet and does both a ping and a curl getting Tasmota status0 information. These were just experiments, I guess I could drop the ping and use just the curl. It's basically a discovery script that lists all active devices on the subnet and adds some extra info if they are Tasmotas.

So I guess the only advantage of the parallel command is if you happen to prefer the cleanliness of the syntax and don't mind installing an extra bit of software. Basically that parallel -j0 --tag single 10.0.0.{1} ::: {1..254} line takes care of the loop and of waiting for the terminations. There are probably also some performance advantages but I didn't look into that, I wasn't too concerned in this case.

When I interrupted work on this, I was trying to avoid all the output that curl throws on the console, I had tried a few of the obvious /dev/null tricks but it didn't work completely; I'm sure there's a solution but I didn't quite reach it in the time invested.