Optimizing the Docker container

[fquirin]

The official SEPIA-Home Docker container is currently not as user friendly as it could be. To properly configure the SEPIA server, the persistent data folder and the core user accounts a tricky procedure is required before the first start, often leading to the following issues:

• All persistent data is stored in one folder, namely $HOME/SEPIA which also contains the entire server. Mounting this folder to a Docker volume has to be done in the right way (create volume, make sure its empty!, mount it properly) or the folder will be empty at start and the server won't work.
• To set up the database and create the core-accounts (admin, assist) with user-defined passwords the container has to be run with a different Docker command first and the user has to use the setup menu via Linux terminal.
• The Nginx proxy setup has some features that usually only work outside of the Docker container like self-signed SSL which depends on the hostname and IP of the non-virtual, external network. Any SSL related stuff should be handled via the Docker host machine not inside the SEPIA container.
• Updating the Docker container doesn't actually update the SEPIA server because all data is in the shared volume, so even the new container will use the old data. If you want to keep your old data you have to run the update script instead.
• Probably some other stuff I forgot.

When SEPIA was first created it wasn't with Docker in mind (the hype wasn't really there yet) and due to Java it wasn't really an advantage to run it as container (too much overhead, more complicated to set up and run). The landscape changed a bit when home-servers/NAS etc. came equipped with the possibility to quickly spin up new Docker containers and besides that the idea of "sandboxing" everything seemed appealing as well.
So here we are and I think it is finally time to fixe the issues mentioned above. Luckily most of this should be straight forward :-). Some ideas:

• Collect all persistent user data in one folder that is initially empty. This should be possible using Linux Symlinks for the database folder, plugins/custom services folder, web-content folder, TTS engines folder and some assistant data including the properties files. Most of the paths should be mentioned inside the backup script.
• The core-accounts could be created automatically using a small setup text-file, placed in the new, empty folder for persistent data.
• Nginx proxy setup should be fixed and user should be noted that he has to handle SSL etc. from Docker host machine
• If all folders are managed properly a server update should be as easy as pulling the new container.

I think this will already greatly enhance the user experience and I'll probably start implementing these changes after the next release.

If you have more ideas feel free to comment below! 🙂

[SHU-red]

Again thank you very much, making up your mind about such things!

As you know i had a few difficulties starting up my "SEPIA-Journey" in Docker.
As i already mentioned i dont feel like an expert user and a few weeks ago (with your help) i could make SEPIA work on my homeserver with some trial and error during installation.

A few thoughts:

• At the moment (and as your documentation in docker hub says) it it necessary to startup the container multiple times with different configurations to get it running -> Target shoud be, to only have one initial configuration to start and thats it
• Maybe you could share a docker-compose config on the docker hub site (i will post my current one below this comment), this is more convenient and poplular (i think), instead of using the standard docker commands
• Also changing the tagging on dockerhub could be useful? -> Im sure you know about the widely used :latest tag to have the possibility to auto-update conatainers (which could for sure lead to problems 😉 )
• Im not an expert but auto-generating docker-containers via github actions is possible (which i don't know much about but i have this on my todo list)
• Due to the fact, that you have more than one service in your docker container which causes distributed config files, symlinks could work (If you can handle those within the bigger SEPIA-context)
• Collect all persistent user data in one folder that is initially empty. -> yes i think this is the right way

Im not sure what else i can provide but

1. Maybe i am mabe to contribute somehow (even if its documentation)
2. I could maybe test some stuff if necessary but im always afraid to break my setup 😅

services:
  sepia:
    container_name: sepia
    depends_on:
      - sepia_stt # Only if sepia_stt is used
    image: sepia/home:v2.6.1_amd64
    restart: unless-stopped
    ports:
      - "20726:20726"
    volumes:
      - /[YOURPATH]/sepia_data:/home/admin/SEPIA

  sepia_stt:
    container_name: sepia_stt
    image: sepia/stt-server:vosk_amd64
    restart: unless-stopped
    ports:
      - "20741:20741"
    volumes:
      - /[YOURPATH]/sepia_data:/home/admin/SEPIA #  Mounted to sepia-home folder

[fquirin]

Maybe you could share a docker-compose config

yes, that looks like a useful tool. I haven't really spent much time with it yet, but it looks easy enough ^^.

changing the tagging on dockerhub could be useful? -> Im sure you know about the widely used :latest tag to have the possibility to auto-update conatainers

I think I never used the 'latest' tag so far because I'm not 100% sure how Docker selects the correct platform (x86/ARM, 32/64bit) in this case and I was too "lazy" to upload each container with version AND latest tag (although I think you can simply define an alias somehow) 🤔

Im not an expert but auto-generating docker-containers via github actions is possible

yes, it looks interesting. I think GitHub is also hosting the containers in this case. I don't know yet how it works though.

Im not sure what else i can provide but ...

The docker-compose YAML looks interesting :-) 👍

[SHU-red]

I think I never used the 'latest' tag so far because I'm not 100% sure how Docker selects the correct platform (x86/ARM, 32/64bit) in this case and I was too "lazy" to upload each container with version AND latest tag (although I think you can simply define an alias somehow) thinking

Thats very easy
Everytime you upload a new image with a tag that already exists, the tag disappears from the old image and is only availalbe on the latest uploaded
Thats how for example the Docker-Container "Watchtower" checks frequently all the images used in the local Docker service, if the image hash from Docker Hub has changed -> If it did change, it downloads the newer one -> So you have some sort of auto-update of your docker container -> Im using it and AFAIK it is very common to do so

So in my case if i upload a new version i always push the same image to times

1. latest
2. e.g. 0.4 (increasing number)

This way the user can choose a fixed version or always being up to date by using latest tag

The correct platform is chosen automatically.
Using $ docker buildx gives you the possibility to create multi-arch-images during a singe build which will then be pushed to docker hub as multi-arch build

But regarding installing/configuring the right packages for buildx, this could be more effort. I personally am not able to get it working to build arm-images on my local machine.

yes, it looks interesting. I think GitHub is also hosting the containers in this case. I don't know yet how it works though.

This solution is maybe very nice because once generated you dont have to care about anything, and it may be easier to use GitHubs environment than your local machine, pointing to the topic before where i mention that i am struggling configuring buildx on my local machine to build arm-images

The docker-compose YAML looks interesting :-) +1

😄

[fquirin]

I have experimented a bit with moving files around and creating symlinks.
Anyone who's interested can checkout the "beta" by grabbing the script and put it in your "SEPIA/scripts" folder, e.g.:

cd $HOME/SEPIA/scripts
wget https://github.com/SEPIA-Framework/sepia-installation-and-setup/raw/dev/sepia-custom-bundle-folder/scripts/create-external-data-folder.sh

Then run: bash create-external-data-folder.sh

It should be pretty robust in restoring folder configurations after updates etc. so one can simply call it again.
Just make sure to create a backup before experimenting ^^

[SHU-red]

I will try to test it!
I will need a few days ...

[fquirin]

I've updated SEPIA-Home and the aarch64 and amd64 Docker containers 😁

[SHU-red]

Lets see when the update arrives due to the automatic watchtower updating ;-)

[fquirin]

🤞 😅
Btw since I decided to risk it and use the new "user data folder" there is a lot of updated Docker info, for example: https://github.com/SEPIA-Framework/sepia-docs/wiki/SEPIA-inside-virtual-environments#update-your-container

[thedolt]

Hi Florian,

I'm working on making the deployment of SEPIA a repeatable process on ARM systems. As such, I've turned to containers. I'm writing a simple bash script to deploy most of the common components I use in my home automation to containers (similar to IOTStack but less generalized). To that end, if there's a sequence of steps that have to happen before I can fully instantiate a SEPIA docker container running as a service, I'd like to get those steps a little more clearly mapped out.

The one question (I'm sure there will be more) that has surfaced during this setup is the mapping of the sepia-home-share volume. I'd like to understand why your documentation says to initially run the SEPIA container with the sepia-home-share volume mapped to /home/admin/sepia-home-data, but the default run command says to map the sepia-home-share volume to /home/admin/SEPIA. I suspect this has something to do with what you're talking about in this post, but I want to see if you could provide some more clarity into why this is sequenced this way so that I understand what needs to happen with that volume as the setup progresses.

Any insights will be greatly appreciated.

Cheers!

[thedolt]

Thank you @fquirin!

This is encouraging information. Thank you for the quick reply!

Cheers,

John H

[fquirin]

The only thing you have to be aware of is that Docker has to be able to copy the data into the shared folder on first start or you will end up with an error message and a single log file in your folder. Docker will do this under 2 conditions:

• Either you properly create an EMPTY shared volume (docker volume create ...)
• Or you define a folder inside the run command that DID NOT YET EXIST before the first run

[thedolt]

If you still see something like /home/admin/SEPIA as Docker volume anywhere please point me to the location and I will review it and fix the path! The one location you've mentioned should be good now ^^.

This section doesn't specifically mention /home/admin/SEPIA, but I wanted to raise awareness on the format of the share name vs it's mount location within the container.
https://github.com/SEPIA-Framework/sepia-docs/wiki/SEPIA-inside-virtual-environments#linux

In this segment, the instructions say to create a share in the user's home folder, /home/[my-user]/sepia-home-share, with the volume name sepia-home-share while the mount location inside the container is /home/admin/sepia-home-data.

There is technically nothing wrong with the instructions because the created volume and the referenced volume on the container are the same. I'm not nitpicking this, either, because I know I can change volume name and on-disk location to whatever I want. I just wanted to make you aware in case you were looking for uniformity.

Cheers,

John H

[fquirin]

I agree this was probably not the best decision 😅. I likely chose this name because I have a habit of naming all Docker volumes "xy-share" 🤔. I guess it is not too late to fix this ^^.

[fquirin]

I've renamed 'sepia-home-share' to 'sepia-home-data' in the Docker info pages, just kept 'sepia-home-share' in the instructions for converting old container to new. I think that makes a lot of sense and helps to distinguish the old and new volume in case users where using pre v2.6.2 versions :-)