Refactoring AWS Authentication across the plugins

[ashwanthkumar]

Not using special environment variables like AWS_USE_IAM_ROLE to enable the IAM Role Access. Using the AWS SDK's AWSCredentialsProviderChain in the following order

• Environment Variable Based
• System Properties Based
• Profile on the credentials file based
• Instance Profile based
• Basic Auth using the Keys from the configuration property explicitly
• Anonymous access (if enabled)

@manojlds @brewkode Feedback please.

[ashwanthkumar]

@haimizrael Looping you in, for your feedback.

[haimizrael]

Though other solutions may exist, we have an use case where this environment-variable trigger was necessary to override AWS access keys set at the system level and authenticate the s3 plug-ins via the instance profile. Specifically, a pipeline task authenticates to aws via access key (giving it rights possibly in a different aws account), while the s3 plugin authenticates via the role.

The main issue is allowing the s3 plugins authenticate with one set of rights to aws while other tasks in the pipeline authenticate with different set of rights. If AWS_USE_IAM_ROLE is removed, I think we'd need a different mechanism - perhaps a variable that holds an AWS role name to assume. If present, the s3 plugin assumes that role to perform the work. Otherwise, if using solely instance profile, you'd have to give all your instance profiles rights to access the s3 artifacts buckets plus other, application, rights, which compromises security by giving too much access rights to all roles.

[ashwanthkumar]

@haimizrael Please help me with few clarifications just so I understand this better

Specifically, a pipeline task authenticates to aws via access key (giving it rights possibly in a different aws account), while the s3 plugin authenticates via the role.

Well according to AWSCredentialsProviderChain, it would pick up the first authentication mechanism that allows access to the required artifact bucket. Assuming access key has fewer permissions than InstanceProfile, we would naturally pick only the instance profile. If latter is true, we would only use aws keys available on the machine (GoCD Server and all agents for material plugin, and only the agents for fetch and publish plugin).

The main issue is allowing the s3 plugins authenticate with one set of rights to aws while other tasks in the pipeline authenticate with different set of rights.

I would really love to know how do you handle this scenario in your current setup? Is it like the AWS keys on the instance has more access permissions while the Instance profile has only GoCD specific access? And you use the AWS_USE_IAM_ROLE environment variable to force use IAM roles and not use AWS keys on ~/.aws/credentials? From the previous quote, I understand this is the case. Please correct me if my understanding is wrong. Assuming the above claim is right, I don't understand how is this approach better from the security standpoint? If someone has access to the machine they might as well use the ~/.aws/credentials to perform whatever actions they want and same holds true for Instance Profile as well isn't?

If AWS_USE_IAM_ROLE is removed, I think we'd need a different mechanism - perhaps a variable that holds an AWS role name to assume.

Is this related to #40 ?

[ashwanthkumar]

@haimizrael @brewkode @manojlds If there are no more comments / concerns, can we merge this?

[haimizrael]

@ashwanthkumar : Apologies for not following up. I somehow missed your update and questions on my post.

My main issue is that, because environment variables (and thus - AWS authentication) can be controlled at the Go Job (and not Go Task) level, without ind s3 plugin-specific environment variables to control the AWS authentication (such as AWS_USE_IAM_ROLE), I cannot easily have in the same Go Job the Fetch S3 Artifacts task use one AWS auth, while the remaining tasks in that Job use different AWS auth.

Our specific situation is that we have different AWS accounts (prod, non-prod, etc.). Go artifacts are stored in a bucket accessible by the owning account, and agents whose primary workload is in a different account need to access this bucket to fetch or push artifacts, while the remaining work is done in another AWS account. While it is technically possible to provide cross-account access to the bucket, due to organizational constraints and the number of Go servers and teams that use them, that is not currently feasible for us. A solution like you mention in #40 would definitely help, as then access can be controlled via a single role (and grant other aws creds to assume it), as opposed to granting resource access to multiple roles and accounts.

In regards to your specific question regarding having both secret/access key and role access via the machine: we apply access/secret keys via encrypted environment variables in Go (not ~/.aws/credentials) such that if one would gain access to the machine, they would only have access to the role associated with the instance profile.

Bottom line: while I do like the simplicity of AWSCredentialsProviderChain for authentication, without an assume-role solution as described in #40, if you merge this PR in we'd have to stay on our fork as we won't be able to function with it in our environment.

[ashwanthkumar]

Bottom line: while I do like the simplicity of AWSCredentialsProviderChain for authentication, without an assume-role solution as described in #40, if you merge this PR in we'd have to stay on our fork as we won't be able to function with it in our environment.

@haimizrael I wanted to lay the foundations for implementing #40 with this PR. Rest assured, we should have #40 also baked in when we're going to make the first release with this PR. I hope that answers the above mentioned concerns?

[haimizrael]

Yes, with #40 in the release, we'd have a way of implementing this in our situation.

Thanks, @ashwanthkumar

[vladistan]

So I finally got the time to work on #40. Just curious what's the status of this PR? Are you planning to merge any of this code into eventually or is it just a dead-end branch? @ashwanthkumar can you chime in on this?

[manojlds]

Closing this in favour of #53