New mle abort subcmd - Clean experiment termination

[RobertTLange]

I would like to have a subcommand that terminates all jobs associated with an experiment and removes all generated files/the trace of it. Otherwise one has to manually use qdel, scancel or gcloud compute instances delete. This could for example be mle abort <experiment_id> or simply mle abort with an additional user Q/A afterwards (check if the status of the experiment is running). A simple procedure could look as follows:

0. Print summary filtered by status being running and get experiment from cmd args or user.
1. Check if experiment_id is in db and status is running. Repeat Q if not.
2. Get job name from single_job_args.job_name in DB.
3. Delete all jobs starting with a job_name. This will depend on the resource.
4. Delete all files in experiment_dir.
5. [Maybe 1. instead] Set the experiment status to aborted in the DB and push it back to GCP.

• Main problem: Grid search experiments launch new jobs based on job termination. How do we circumvent this?
• Potential Solution: Update database between grid search batches and check if the status was set to aborted. If no: Update batch counter in database. If yes: Stop launching new jobs. Break out of hyperparameter run. This also has the advantage that we can also show the current batch iteration in mle monitor.

Also allow user to choose between termination via experiment config .yaml and experiment_id.

Note: Give credit to Tudor's Liftoff package.

[RobertTLange]

It would be great to have a keyboard interrupt wrapper that cleans up the protocol/VM instances. Have a look at this thread: https://stackoverflow.com/questions/1187970/how-to-exit-from-python-without-traceback