Openstack: support for anti-affinity server groups.

[flashvoid]

Hello,

We made this feature for one of our clients and interested if it would be welcomed upstream.

The goal is to provide better results for cpu hungry applications on openstack clouds with cpu overcommit.

[riccardomurri]

Is it possible that the out_of_capacity message gets localized and we miss it? Is there an error code, exception class, or other more invariant means of checking what the issue is?

[flashvoid]

This is a very good point, unfortunately, there doesn't seem to be any way to assert the reason of an error other than the error message. The call is async, there is no distinct error code and error itself is similar to one when an entire cloud runs out of resources.

Maybe it makes sense to drop second conditional and just leave as it is. The logic would be to stop growing the group after first instance that fails to start, regardless of a reason.

[flashvoid]

Just made an extra test, thats what openstack tells when scheduler fails to schedule an instance due to anti-affinity.

ipdb> pp vm.fault
{u'code': 500,
 u'created': u'2018-08-08T04:22:56Z',
 u'message': u'No valid host was found. There are not enough hosts available.'}

[riccardomurri]

This branch of the if misses the with ...__node_start_lock

[flashvoid]

No that was on purpose.
Cluster.start() allows two modes of node creation, sync and async. Async isn't compatible with aaf groups because we need feedback from the previous instance before making a decision about next.

So effectively, anti_affinity_group_prefix implies -p 1, and this line represents backward compatibility for people who want async and do not want to use aaf groups.

P.S. I would, however, make a point that async in case of openstack doesn't save much time, create_instance API is async by nature so -p 1 wouldn't make much difference.

[riccardomurri]

Will this work for any OpenStack user? Or does it require admin access?

[flashvoid]

No admin privileges needed in our cloud and as far as I can tell it is a default behavior to allow regular users to use this feature.

[riccardomurri]

Is the "1" really a constant here? If yes, why not make it part of self.middle_token?

[flashvoid]

It is a constant and it isn't a part of self.middle_token because former is used in self._list_anti_affinity_groups() for substring matching.

[riccardomurri]

Very minor point: I would prefer if self.middle_token was self._middle_token so it does not look like it's part of the public API / parameters of the class. (I know the ElastiCluster sources are inconsistent on this :-))

[flashvoid]

ok, will do

[riccardomurri]

Thank you very much for this contibution! It would be fantastic to have it merged upstream; I think it would be extremely useful to whoever is running on mixed-workload oversubscribed clouds.

The code looks good to me; I have one important question: will it work for any OpenStack user or do anti-affinity groups require some kind of admin privileges?

I have noted a few other minor points in the sources; great if you have time to fix (or talk back) yourself, otherwise I can take care of that when merging.

Thanks again for taking the time to contribute!

[riccardomurri]

Thank you very much for following up! I'll be back on Friday: then I'll test the code and merge.

[riccardomurri]

So the code looks fine and doesn't break existing functionality :-) I cannot really test the AAF feature, as it is not supported by our local OpenStack; I'll take your word that it works :-)

A couple more questions related to AAF:

1. at the moment, the AAF prefix is set cloud-wide; does it make sense to set it cluster-wide? (i.e., different clusters on the same cloud use different prefixes) Otherwise, it seems to me that stopping a cluster will could tear down the AAF groups used for other clusters.

2. I still don't get the prefix + middle token + number thing: I understand that you want to list all AAF groups "belonging" to an ElastiCluster session (hence the need for a unique substring, which at the moment is played by prefix+middle), and that you need to generate multiple AAF groups (hence the addition of the final number). But then why is the middle token constant? if it's constant, it plays no uniqueness role and can be omitted; if it serves the purpose of distinguishing ElastiCluster-generated AAF groups from ones that users may legitimately create on their own, then we should probably chose something more specific than .number. (e.g., _elasticluster_ to indicate origin, or some randomly-generated string to minimize probability of collisions).

[flashvoid]

@riccardomurri

I cannot really test the AAF feature, as it is not supported by our local OpenStack; I'll take your word that it works :-)

I will be happy to set up an account for you on our public cloud so you could test things. Once we work out the reviews here.

then we should probably choose something more specific than .number.

I agree, will fix the middle token to be _elasticloud_

at the moment, the AAF prefix is set cloud-wide; does it make sense to set it cluster-wide?

Well, I was addressing specific use-case and might have missed the bigger picture (after all that what reviews are for :-) ). What would be the desired way to pass cluster-wide configuration down to the openstack provider? would you recommend passing it with the Node object?

[riccardomurri]

Hello again!

Sorry this had completely fallen off my radar -- I have restructured the code a bit to work with parallel starting of instances, and for using a single configuration item (boolean use_anti_affinity_groups):

https://github.com/gc3-uzh-ch/elasticluster/compare/master...riccardomurri:pr/%23577?expand=1

I think we still need to record the AAF group names somehow; I understand "delete all" will only be performed by OpenStack if all server groups are empty, but I think it would be cleaner if every instance just cleaned up its own server group.

[riccardomurri]

I think it would be cleaner if every instance just cleaned up its own server group.

This is now implemented in my pr/#577 branch, on top of your commits. Would you have time to test it? I do not have access to an OpenStack installation with AAF groups.