latent worker machine for EC2 Mac instances (with 24hours minimum runtime)

[hmoffatt]

I would like to have a worker running on a machine which is suspended and resumed on demand. I see there is some code from @p12tic in machine/ which looks relevant, like GenericLatentMachine, but I couldn't find any documentation or concrete examples of how to use it.

What I'd actually like to do is have an EC2 instance which is suspended and resumed as required. The latent workers provided with Buildbot all create new instances from a template (eg EC2 AMI) each time they start, but this doesn't suit an AWS EC2 Mac instance, which has a minimum runtime of 24 hours. So I would like to create an EC2 instance myself but have it shutdown when not required to save costs.

[tardyp]

Hello, looks like you are needed a EC2LatentWorker, not a GenericLatentMachine
https://docs.buildbot.net/latest/manual/configuration/workers-ec2.html

most of the infrastructure is there for authentication and stuff
You would need to customize the code to reuse the instance if really needed. I don't think this will be very difficult

That said, I am not sure if reusing the instance will really matter actually even for Mac instances. I would tweak the latent worker build wait timeout in order to keep the machine alive for 24 hours
http://docs.buildbot.net/current/manual/configuration/workers.html#common-latent-workers-options

This means the machine will stay up until there is 24 hours of inactivity.

[hmoffatt]

As I read the EC2 documentation, every new Mac instance needs to run for at least 24 hours to meet the license conditions. So if I want to run a nightly build I would have instances running permanently. But I could reuse the same instance forever and keep it suspended when not in use.

Hence my question, can I use the code for a latent machine and is there any documentation/example?

Alternatively I might use a Mac on premises but I would like to wake it up when required. The latent machine code hints at this approach too.

[hmoffatt]

Are there any concrete examples of using the LocalWakeAction, LocalWOLAction, RemoteSshWakeAction etc that are defined in machine/generic.py?

There's an IMachineAction interface that these classes implement but no consumers of them that I see.

[p12tic]

I will dig the configuration that I had working some time ago. IIRC not all changes needed for machines to run were contributed into Buildbot, this would be good chance to finish it.

[hmoffatt]

Would it be possible to have a simple latent worker that took a shell command to run to start and stop the worker?

With that one could wake another machine and ssh in to start/stop a service, or start a VM or use the awscli to start/stop an existing ec2 instance etc.

[p12tic]

I'll explain the difference between machines and workers.

One thing to keep in mind is that in buildbot the actual, physical worker that connects to the master is always the same. It always attempts to connect to master and while it's possible to shut it down, it's permanent as worker would need to be restarted by other means and thus not useful. All worker lifetime management happens by other means - starting/shutting down containers, VMs, EC2 machines and so on.

The worker management happens on master side. Regular workers (Worker class) just wait for some worker to connect. Latent workers do some additional setup to bring a single worker to life for the duration of a build. EC2LatentWorker starts EC2 machine, Libvirt workers starts a VM and so on. The important thing is that a single latent worker manages a single physical worker. So it's not possible to bring up a EC2 machine with e.g. 3 workers just by using EC2LatentWorker.

This limitation is solved by machines. These just represent as set of scripts that can start up/shut down some physical thing on which one or more workers can run. So e.g. you can in theory setup a beefy and power hungry machine with 10 workers, suspend it and then the master would wake it up whenever a build for any of these workers comes up.

So for your use case machine is exact fit, just like I said in a previous comment I check what I have and port to upstream buildbot.

[hmoffatt]

Thanks for your detailed reply, this is very helpful in understanding the internal design regarding machines and latent workers.

[sananand007]

What was the conclusion here ? @hmoffatt , were you able to work with BuildBot, in adding EC2 machines , I have a similar use case and would like to check

[hmoffatt]

HI @sananand007 , no I have not solved this and am running a dedicated Mac worker on my own hardware instead.

[sananand007]

Also in general, How do we add a generic LatentWorker any remote machine after opening the TCP port ? Is there any knowledge to share in that respect ?