picroft never updates skills

[fermulator]

Describe the bug
Mycroft installed via picroft, seems to never update/upgrade the skills that are installed (neither from default nor from user-added subsequently).

Noticed and discussed: https://chat.mycroft.ai/community/pl/khrqtqa97bdcdg3gn4513arrjo

To Reproduce
Steps to reproduce the behavior:

1. Install picroft https://mycroft-ai.gitbook.io/docs/using-mycroft-ai/get-mycroft/picroft
2. Use mycroft, install some additional skills if you like
3. Over the course of weeks, months, (a year), ... periodically on restart & login, the auto_run.sh script will run, updating mycroft-core
4. But eventually we will notice that skills are NEVER updated :(

Expected behavior

1. mycroft skills need to be updated step-wise with mycroft-core
2. ideally their git branches are kept in sync with core (unless the user changes one? not sure how to detect)

Log files
Here's a sample from my system after a year of use

(.venv) pi@picroft:~/mycroft-core $ git status
On branch master
Your branch is up to date with 'origin/master'.

(.venv) pi@picroft:/opt/mycroft/skills $ for file in $(ls -1); do cd $file; echo $file; git status  | grep branch; cd ..; done
cocktails.forslund
On branch master
Your branch is up to date with 'origin/master'.
dismissal-skill.chancencounter
On branch master
Your branch is up to date with 'origin/master'.
fallback-query.mycroftai
On branch 19.08
Your branch is ahead of 'origin/19.08' by 16 commits.
fallback-unknown.mycroftai
On branch 19.08
Your branch is ahead of 'origin/19.08' by 4 commits.
fallback-wolfram-alpha.mycroftai
On branch 19.08
Your branch is up to date with 'origin/19.08'.
homeassistant.mycroftai
On branch 20.02
Your branch is behind 'origin/20.02' by 18 commits, and can be fast-forwarded.
  (use "git pull" to update your local branch)
mycroft-alarm.mycroftai
On branch 19.08
Your branch is ahead of 'origin/19.08' by 40 commits.
mycroft-audio-record.mycroftai
On branch 19.08
Your branch is ahead of 'origin/19.08' by 9 commits.
mycroft-configuration.mycroftai
On branch 19.08
Your branch is ahead of 'origin/19.08' by 5 commits.
mycroft-date-time.mycroftai
On branch 19.08
Your branch is ahead of 'origin/19.08' by 36 commits.
mycroft-fallback-duck-duck-go.mycroftai
On branch 19.08
Your branch is up to date with 'origin/19.08'.
mycroft-hello-world.mycroftai
On branch 19.08
Your branch is ahead of 'origin/19.08' by 2 commits.
mycroft-installer.mycroftai
On branch 19.08
Your branch is ahead of 'origin/19.08' by 12 commits.
mycroft-ip.mycroftai
On branch 19.08
Your branch is ahead of 'origin/19.08' by 4 commits.
mycroft-joke.mycroftai
On branch 19.08
Your branch is ahead of 'origin/19.08' by 8 commits.
mycroft-naptime.mycroftai
On branch 19.08
Your branch is ahead of 'origin/19.08' by 6 commits.
mycroft-npr-news.mycroftai
On branch 19.08
Your branch and 'origin/19.08' have diverged,
  (use "git pull" to merge the remote branch into yours)
mycroft-pairing.mycroftai
On branch 19.08
Your branch is ahead of 'origin/19.08' by 5 commits.
mycroft-personal.mycroftai
On branch 19.08
Your branch is ahead of 'origin/19.08' by 1 commit.
mycroft-playback-control.mycroftai
On branch 19.08
Your branch is ahead of 'origin/19.08' by 9 commits.
mycroft-reminder.mycroftai
On branch 19.08
Your branch is ahead of 'origin/19.08' by 2 commits.
mycroft-singing.mycroftai
On branch 19.08
Your branch is ahead of 'origin/19.08' by 11 commits.
mycroft-speak.mycroftai
On branch 19.08
Your branch is ahead of 'origin/19.08' by 1 commit.
mycroft-spelling.mycroftai
On branch 19.08
Your branch is ahead of 'origin/19.08' by 1 commit.
mycroft-spotify.forslund
On branch 20.02
Your branch is behind 'origin/20.02' by 1 commit, and can be fast-forwarded.
  (use "git pull" to update your local branch)
mycroft-stock.mycroftai
On branch 19.08
Your branch is ahead of 'origin/19.08' by 28 commits.
mycroft-stop.mycroftai
On branch 19.08
Your branch is ahead of 'origin/19.08' by 2 commits.
mycroft-support-helper.mycroftai
On branch 19.08
Your branch is behind 'origin/19.08' by 2 commits, and can be fast-forwarded.
  (use "git pull" to update your local branch)
mycroft-timer.mycroftai
On branch 19.08
Your branch is ahead of 'origin/19.08' by 61 commits.
mycroft-version-checker.mycroftai
On branch 19.08
Your branch is ahead of 'origin/19.08' by 4 commits.
mycroft-volume.mycroftai
On branch 19.08
Your branch is ahead of 'origin/19.08' by 12 commits.
mycroft-weather.mycroftai
On branch 19.08
Your branch is ahead of 'origin/19.08' by 52 commits.
mycroft-wiki.mycroftai
On branch 19.08
Your branch is ahead of 'origin/19.08' by 19 commits.
remember.luke5sky
On branch master
Your branch is up to date with 'origin/master'.
skill-finished-booting.zelmon64
On branch master
Your branch is up to date with 'origin/master'.
speedtest.luke5sky
On branch master
Your branch is up to date with 'origin/master'.
```
**Environment (please complete the following information):**
```
 $ ./get-mycroft-community-spec-table.sh 

   DATE, PI VERSION, MYCROFT CORE VERSION, PICROFT VERSION, PI HW THERMAL MANAGEMENT, PI HW CASE, CPU TEMP, GPU TEMP, PI THROTTLING, LOAD AVGS, USER
   "2020-11-16", "Raspberry Pi 3 Model B Plus Rev 1.3", "20.8.0", "Buster Keaton - Pork Pi"
```

**Additional context**

We have a "suspicion" that this is to blame for randomly losing settings in skills (TBD)

[fermulator]

Now I'm not sure how this happened, if it is a user change/error I did at some point, or if this is affecting everyone.

I CAN say, it is highly unlikely that I went into EVERY skill, and changed the branch to a specific target like is seen above. I DID do it for one or two skills for testing, but not all of them.

To get out of this, I did:

for file in $(ls -1); do cd $file; echo $file; git checkout master; git pull origin master; cd ..; done

, then restarted

EDIT: this might be a bad ❌ idea ... after doing so mycroft stopped working

[fermulator]

I found https://mycroft.ai/blog/skill-branching-18-02/ .. but this is super old, not sure what the current strategy is.

[forslund]

Hi @fermulator the mycroft skills uses branches for the different versions. typically the latest version is on the branch named the same thing as the current major version of mycroft-core. So all mycroft skills has a branch called 20.08 which is the branch latest development. while 20.02 will point to the latest commit of the skill made for the 20.2.x release of mycroft-core.

The information shown is a bit misleading...the way msm updates the skills is via git merge --ff-only SHA, where SHA is the commit sha referred to by mycroft-skills. so the branch name is actually never changed from the one used when the skill was checked out (but newer commits are applied to it).

We should probably just have used main/master as latest/current version branch and do branches for the older mycroft-core versions and push fixes to those branches if needed (like we do now)

Checking out master on the mycroft-core skills will likely return them to a very old state

[fermulator]

I see - indeed - not using a consistent branch and instead fast forwarding to SHA commit is definitely confusing :)
Are there long term plans here to address/improve? Or will we never correct this?
(the title therefore may not accurately reflect the reality)

[forslund]

Not sure where the discussions in regards to this issue. There has been some discussion of alternate strategies, for example moving from git to tar-balls. OVOS skill installer has some support for this.

[JarbasAl]

OVOS skill manager uses tarballs and not git, any branch can be selected and an update simply replaces previous install, but we disagree with automatic skill updates, that will always require a user action

either way OSM was made due to creative differences with mycroft-core, so that will never be the solution for this issue since MSM will continue doing it's own thing and it's baked into core

there is ongoing work on a OSM skill to use it with mycroft, that will replace the official skill installer skill and require https://github.com/OpenVoiceOS/disable-msm-dummy-repo

keep in mind OSM is in alpha version, that sums up the state of things in regards to OVOS skill manager / OVOS skill installer

[krisgesling]

My thinking is that a Skill update should be able to overwrite the entire directory of that Skill, as Skill settings and Skills data are now stored elsewhere. Previously the Skill settings - settings.json was stored in the directory of the Skill.

So at the moment, my plan was to use tarballs, and an update would simply replace the old Skill with the new one. Whether that's using OSM or updating MSM I don't know as I haven't looked into them yet. However we definitely do want automatic Skill updates, so on the surface I'd assume updating MSM.

[fermulator]

what's the latest/greatest here I should try and check?

[krisgesling]

Hey there, we haven't done any work on the Skills management side of things. We've got some more fundamental changes to make before we get back to this sorry.