AWS Glue: BIGQUERY connection type support

[ksauzz]

Describe the bug

aws-sdk-go doesn't accept BIGQUERY as connection type because API specification file seems too old.

According to the blog post, AWS Glue has suppported BIGQUERY since Oct 2023, but the API version is 2017-03-31.

Expected Behavior

aws-sdk-go accepts BIGQUERY as connection type.

Current Behavior

Error: expected connection_type to be one of ["JDBC" "SFTP" "MONGODB" "KAFKA" "NETWORK" "MARKETPLACE" "CUSTOM"], got BIGQUERY

Reproduction Steps

See API Model.
https://github.com/aws/aws-sdk-go/blob/main/models/apis/glue/2017-03-31/api-2.json#L4942-L4953

Possible Solution

Update models/apis/glue/2017-03-31 to the latest one, and run make generate

Additional Information/Context

I couldn't find any newer glue API model in any SDK repositories.

• https://github.com/boto/botocore/tree/develop/botocore/data/glue/2017-03-31
• https://github.com/aws/aws-sdk-go/tree/main/models/apis/glue/2017-03-31
• https://github.com/aws/aws-sdk-go-v2/blob/main/codegen/sdk-codegen/aws-models/glue.json
• https://github.com/aws/aws-sdk-java-v2/blob/master/services/glue/src/main/resources/codegen-resources/service-2.json
• https://github.com/aws/aws-sdk-php/blob/master/src/data/glue/2017-03-31/api-2.json
• https://github.com/aws/aws-sdk-ruby/tree/version-3/apis/glue/2017-03-31
• https://github.com/aws/aws-sdk-js-v3/blob/main/codegen/sdk-codegen/aws-models/glue.json

SDK version used

1.51.31

Environment details (Version of Go (go version)? OS name and version, etc.)

all environment.

[ksauzz]

Our team decided to use spark-bigquery-connector in AWS Glue instead, so this issue is no longer necessary for us. Feel free to close this if no one has interest.

[RanVaknin]

Hi @ksauzz ,

The SDK is code generated from the API model of each service. In this case, the source of truth for the enum type the SDK uses is the internal API model for glue which does not specify BIGQUERY as a valid enum value for connectionType.

According to the blog post, AWS Glue has suppported BIGQUERY since Oct 2023, but the API version is 2017-03-31.

The API version date you see here is when this major version was released, and is not related to when it was last updated. Since AFAIK Glue only has one major version supported, 2017-03-31 is when this major version was released.

From reading the blogpost I can see that this is indeed supported, but from an API operation perspective I don't see that BIGQUERY is mentioned anywhere under ConnectionType, not in the blogpost or any other service docs. To be clear, you might be 100% correct and the Glue service team simply did not update their API model to include this newly released type and we might need to engage with them to fix it.

I will reach out to confirm with the Glue team if this needs to be added and if so I'll have them update their model to include it.

Thanks,
Ran~

[ksauzz]

Thank you for the explanation in detail.

[RanVaknin]

Hi there,

I was not able to get a confirmation from the Glue team on whether this is the intended value for this enum or whether this value has service side support to begin with.

If anyone is running into this issue in the future, please open a separate issue on our v2 SDK repository issue queue.

Thanks,
Ran~

[github-actions]

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[ksauzz]

I think some users want to manage BIGQUERY connection type through terraform involving aws-sdk-go v1, as well as v2 someday. So even without updating API model, just removing client-side validation for the enum would help them.

Thanks.

hashicorp/terraform-provider-aws#34575