aws_ebs_volume fails to create encrypted KMS EBS volumes

[astral303]

Cookbook version

7.1.2

Chef-client version

12.21.21

Platform Details

Amazon Linux 2017.03, AWS

Scenario:

Attempting to create an encrypted EBS volume using a specific KMS key ID.

Steps to Reproduce:

Attempt to create an EBS volume using a resource similar to:

aws_ebs_volume 'ebs0' do
  size size_gigs
  device device_id
  delete_on_termination false
  volume_type 'gp2'
  encrypted true
  kms_key_id 'arn:aws:kms:...'
  action [:create, :attach]
end

Expected Result:

I expect the EBS volume to be successfully created and attached.

Actual Result:

Creation fails with:

   Aws::EC2::Errors::InvalidVolumeNotFound
   ---------------------------------------
   The volume 'vol-1118028b1c1a11ff1' does not exist.

   Cookbook Trace:
   ---------------
   /tmp/kitchen/cache/cookbooks/aws/resources/ebs_volume.rb:166:in `volume_by_id'
   /tmp/kitchen/cache/cookbooks/aws/resources/ebs_volume.rb:217:in `block (2 levels) in create_volume'
   /tmp/kitchen/cache/cookbooks/aws/resources/ebs_volume.rb:216:in `loop'
   /tmp/kitchen/cache/cookbooks/aws/resources/ebs_volume.rb:216:in `block in create_volume'
   /tmp/kitchen/cache/cookbooks/aws/resources/ebs_volume.rb:215:in `create_volume'
   /tmp/kitchen/cache/cookbooks/aws/resources/ebs_volume.rb:57:in `block (2 levels) in class_from_file'
   /tmp/kitchen/cache/cookbooks/aws/resources/ebs_volume.rb:56:in `block in class_from_file'

More log detail is here: https://gist.github.com/astral303/5f6b8720489d2e0cf9ed2a59dc8c7a54

NOTE: This works fine if I turn off encryption.

Current diagnosis:

It seems that the AWS API is behaving such that if an encrypted EBS volume is created, then it is not at all visible in the volumes list (not even as 'creating'!) for 5 to 30 seconds, and only then does it show up as available.

During this "unavailability" period, the timeout loop ends up checking for the volume and ec2.describe_volumes throws InvalidVolumeNotFound.

Suggested fix is to swallow InvalidVolumeNotFound during create_volume Timeout block, such that if volume_by_id call throws this, the waiting continues.

[glajchs]

We think we know what went wrong now. The KMS key we were using to create the EBS volume did not permissions to be created by the AWS user that chef was using. Unhelpfully, the AWS API returns a successful volume object when this happens, but it seems that in the next couple of seconds, the AWS backend realizes that you've requested a volume to be encrypted in a way that you don't have permission to do, and it just deletes the EBS volume automatically, as if it never (briefly) existed.

In lieu of the AWS EBS create API returning something useful, maybe we can add a blurb to the error message thrown to include this possibility (when KMS key is present in the options)?

[iennae]

That sounds like a great enhancement. Would you be open to submitting a PR? Thanks for following up with more details about what happened.