Create network resources for AWS Local Zones and Wavelength Zones

[mtulio]

/kind feature

Describe the solution you'd like:

Support of creation network infrastructure, subnets, and gateways, required for AWS
Local Zones and Wavelength zones, allowing users to create compute nodes (non-control
plane) specifying those subnets.

AWS Local Zones allow end-users to deploy cloud resources closer to the metropolitan
regions and offer a limited resource, like Instance Type, Block Storage, Application Load
Balancers, Nat Gateways etc.
The network infrastructure required to create nodes in Local Zones are:

• public and private subnets,
• public subnet associated with a route table with Internet Gateway as a default router
• private subnet associated with a route table with a valid Nat Gateway*
• *Currently, Nat Gateway is not globally supported by Local Zone locations, and currently there
  is not API to query if it is offered in the zone.

AWS Wavelength zones are infrastructure running in carrier infrastructure. It also have
limitations of resources offered in those locations.

Network resources required for Wavelength zones:

• Carrier Gateway attached to the VPC
• public and private subnets
• public "carrier" route table with the default route entry targeting the Carrier Gateway
• private route table with the default route entry pointing to a valid Nat Gateway*
• public subnet associated with the public "carrier" route table
• private subnet associated with the private route table
• *Same Nat GW limitation as Local Zone.

Not Goal:

• Create control plane nodes, or required resources, in remote zones.
• Compute/Machine resource creation.

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Currently, OpenShift creates the network resources required to deploy only compute
nodes in AWS Local Zones and Wavelength zones, and we would like to use Cluster API to
deploy those resources. Machines (compute nodes) are then
created, outside Cluster API.

Considering the limitation of network-based load balancers, we used to recommend
to the users to use the AWS Application Load Balancer controller to
create ingress traffic into those zones (day-2).

For the gateway for private subnets, we are reusing the same Nat Gateway for the
Parent Zone (zone attribute), when it exists, otherwise, the
"default" private route table will be used to associate the zone in the remote location.
If possible, it would be nice to deploy Nat Gateway in the location, when supported
(currently, only a few Local Zone locations support it), otherwise, use some strategy like that.

For the gateway for public subnets in Wavelength Zones, we are creating a single
"public carrier" route table, with the default route entry to the Carrier Gateway,
previously created and associated with the VPC,

For more details feel free to read the enhancement with more details here.

The suggested Local Zones workflow looks like this:

• User creates the Subnets (spec.Network.Subnets) specifying all required zones to
  create the cluster, including Local Zone or Wavelength Zones, in the existing API
  field SubnetSpec.AvailabilityZone - keeping the valid placeholder ID to
  teach Cluster API to create subnets.

  zones := []string{"us-east-1a", "us-east-1b", "us-east-1c", "us-east-1-nyc-1a"}	
  for _, zone := range zones {
    cluster.Spec.NetworkSpec.Subnets = append(cluster.Spec.NetworkSpec.Subnets, capa.SubnetSpec{
        AvailabilityZone: zone,
        CidrBlock:        privateCIDRs[idxCIDR].String(),
	ID:               fmt.Sprintf("%s-subnet-private-%s", infraID, zone),
	IsPublic:         false,
	})
    cluster.Spec.NetworkSpec.Subnets = append(cluster.Spec.NetworkSpec.Subnets, capa.SubnetSpec{
        AvailabilityZone: zone,
	CidrBlock:        publicCIDRs[idxCIDR].String(),
	ID:               fmt.Sprintf("%s-subnet-public-%s", infraID, zone),
	IsPublic:         true,
    })
    idxCIDR++
  }

• Network reconciliation creates the Carrier Gateway, attaching to the VPC
• Subnet reconciliation creates the public and private subnets into those zones,
  then discovers the zone attributes* for the subnet's zone
• Subnets prevent returning subnets created in Wavelength Zone locations
  to the regular flow, preventing issues in the controllers consuming the standard flow
• Route reconciliation check whether the public subnet "is Wavelength Zone", and find the
  appropriate route and gateway for public subnets in Carrier Gateways
• Route reconciliation check whether the subnet "is Wavelength Zone", and find the
  appropriate gateway for private subnets
• Regular cluster creation flow

*Required API changes for the proposed workflow:

• Add the following fields to the SubnetSpec:

	// ZoneType defines a zone type for this subnet.
	ZoneType *string `json:"zoneType,omitempty"`

	// ParentZoneName defines a parent zone name for the zone that the subnet is created,
	// when applied. Available only in zone types local-zone or wavelength-zone.
	ParentZoneName *string `json:"parentZoneName,omitempty"`

The suggested Wavelength Zones workflow looks like this:

• User creates the Subnets (spec.Network.Subnets) specifying all required zones to
  create the cluster, including Local Zone or Wavelength Zones, in the existing API
  field SubnetSpec.AvailabilityZone - keeping the valid placeholder ID to
  teach Cluster API to create subnets.

  zones := []string{"us-east-1a", "us-east-1b", "us-east-1c", "us-east-1-wl1-nyc-wlz-1"}
...

• Subnet reconciliation creates the public and private subnets into those zones,
  then discovers the zone attributes* for the subnet's zone
• Subnets prevent returning subnets created in Wavelength Zone locations
  to the regular flow, preventing issues in the controllers consuming the standard flow
• Route reconciliation check whether the subnet "is Local Zone", and find the
  appropriate gateway for private subnets
• Regular cluster creation flow

*Required API changes for the proposed workflow:

• Add the following fields to the VPCSpec:

	// CarrierGatewayID is the id of the Carrier internet gateway associated with the VPC.
	// +optional
	CarrierGatewayID *string `json:"carrierGatewayId,omitempty"`

[k8s-ci-robot]

This issue is currently awaiting triage.

If CAPA/CAPI contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.