Chaveiro component manager

[TrystanLea]

See original pull request: #1732 by @chaveiro
This pull request fixes the conflicts that arose from my original partial merge.
Review still in progress

[TrystanLea]

Hello @chaveiro

I'm holding off on merging this for now as I would really prefer to only maintain one way of installing, updating, uninstalling modules rather than having both the script calls and the php implementation to maintain. Your php approach is a good suggestion and I will definitely consider it carefully when I bring my focus back to this side of emoncms development. I have merged everything else and given you a pull request here to fix the conflicts that were created https://github.com/chaveiro/emoncms/pull/3.

A couple of questions I have that I'm still thinking about are:

• Im not sure if giving the www-data user write permissions to the module directories is a good idea? Is that normal for other projects e.g wordpress, drupal? What are the implications of this that we need to think about?

• Some modules require running an install.sh script, the php implementation so far suggests that the user runs this manually if required. This is not ideal as it would be good if the full install completes as per install script requirements.

Id say that my general feeling on this, even though I do quite like your php implementation, is that the EmonScripts install scripts approach is there already to give users a consistent way of installing the full emoncms stack, reusing that code from within emoncms for installing and updating components is an advantage as it avoids code duplication and maintaining multiple versions of the same thing. To add this additional code to support environments without the ability to run the EmonScripts is Im not sure worth it. Likewise with things like the option to use emoncms without redis. There are cheap VPS services out there now that can provide all the requirements that emoncms needs, Emoncms is free and development time is limited , I would really prefer to target a specific environment and have it work well in that environment, rather than try to maintain in perpetuity support for a lot of different environments.

[chaveiro]

Hi Trystan,

Im not sure if giving the www-data user write
permissions to the module directories

The approuch is to have a 'emoncms' user owning the files and running httpd so not to be www-data that may be running other websites that in case of a security compromise all is restricted to that user only.

Some modules require running an install.sh script

Some may require and some don't, usualy the ones that require are not related to the Web interface but are the core backend services, not possible to install on shared servers anyway. Even for those, php can run sh scripts with the right permissions. Maybe just need to warn the user when it can't be done due to permitions restrictions.
The end goal is to have a self service update area that works for most schenarios on this product.

Emoncms is free and development time is limited , I would really prefer to target a specific environment and have it work well in that environment

Wouldn't be better to think on an unique solution that targets the many environments in decrement of a solution that targets only one?

If we look at current trends, shared hosting is not popular anymore (but still the cheapest) future is aligned with automation and having docker images as virtualization.
Have a core docker emoncms image (with apache and php), mysql and redis as other images and plugins installed on a shared volume via the core image may make more sense for vps support and even raspberry deployment. Decoupling the host OS from the application.

See https://hub.docker.com/_/wordpress for inspiration.

So you update core docker image and then install / update plugins via the admin interface.

[chaveiro]

As a side note, do you have statistics of how many users are using raspberrypi, shared hosting, emoncms.org or vps currently?
That may help decide on the support to prioritise.

[TrystanLea]

Thanks @chaveiro Yes I think docker would be the other environment that would be good to target. My understanding from chatting to others is that the application should not really be able to modify the components installed in a docker image, it should be modified from the docker file itself.. it would mean of course restricting all component install, update, uninstall actions on the docker image.. Im not convinced that we have to adhere to this principle, if it is a principle at all? Are you familiar with how wordpress handles plugins within it's docker container? I guess it must allow modification of the installation?

[TrystanLea]

I don't unfortunately have stats as to how many users use emoncms in different environments, all I can say is that we havent actively tried to support shared, wamp, mamp beyond those initial install guide written maybe 6-7 years ago, and when users ask about support for issues on custom installs we generally suggest installing using EmonScripts as there's always a so many different variables that can affect an installation otherwise.

My guess would be:

1. Raspberrypi environment (emonSD image built using EmonScripts)
2. Emoncms.org
3. Custom install on VPS
4. Install using EmonScripts on RaspberryPi or VPS
5. Docker
6. Shared hosting
7. WAMP & MAMP

[TrystanLea]

Some may require and some don't, usualy the ones that require are not related to the Web interface but are the core backend services, not possible to install on shared servers anyway.

Handling these background services is more complicated on docker as well, from my understanding, in principle each service should have it's own container, but it does all get quite complicated fast when you need a service for emoncms_mqtt, feedwriter, demandshaper, service-runner etc ... and then the web frontend is designed to work with services running as systemd which is not usually an option in a docker container.. or at least in the containers we've developed so far.

Is there a better way (or at least more practical way) to run these background scripts and services within a docker environment and then manage them from the frontend I wonder?

[TrystanLea]

I probably need to get some time to look at the docker side of things again...

[chaveiro]

Are you familiar with how wordpress handles plugins within it's docker container?

Not familiar but a google search helped with this: https://stackoverflow.com/questions/50119409/how-to-update-wordpress-on-docker
They have a core installation and the plugins are installed on the host via shared volume.
Seems a nice way.
You install the docker image once, then the updates goes to a volume (shared folder on the host). If the host volume files are deleted it reverts to the original version on the image. Of course new images can be updated also (manually) when a major release comes out.

Is there a better way (or at least more practical way) to run these background scripts and services within a docker environment and then manage them from the frontend I wonder?

Two option for docker interconnection: network (mysql, redis, apis) or shared files on the host (volumes)

[TrystanLea]

Thanks @chaveiro ok I think the next step for me is to spend some more time looking into the emoncms docker image and to think how best to handle things in things in that environment. Then with the result of that research come back to this pull request.
Thanks for your help!