Improve e2e testing

[sbueringer]

Improve e2e tests

We compared CAPV govmomi & supervisor e2e tests with core CAPI e2e tests.
As a result we want to make the following improvements.

Adding e2e tests:

• P0: Add v1.30 => v1.31 upgrade e2e test + check ci-latest (supervisor+govmomi) (@sbueringer)
  • PR: CAPV: Add 1.31 tests kubernetes/test-infra#32610
  • PR: 📖 Document Kubernetes bump #2999
  • Plan is to add more and more upgrades jobs for new Kubernetes versions until we have the same test coverage as core CAPI.
• P0: Add test "When testing ClusterClass changes [ClusterClass]" (supervisor) (@zhanggbj)
  • PR: 🌱 Fill up supervisor e2e test - ClusterClass changes #3011
• P1: Add test "Ensure OwnerReferences and Finalizers are resilient" (supervisor) (@fabriziopandini)
  • Should be added at the end of the supervisor quickstart test
  • Depends on 🌱 Improve E2E ValidateFinalizers and ValidateOwnerRef cluster-api#10693 to merge in CAPI / CAPI Bump
• P1: Also use ValidateResourceVersionStable func (🌱 Ensure resourceVersions are stable cluster-api#10530) to ownerRef & finalizer tests (supervisor+govmomi) (@fabriziopandini)
  • Depends on bump to CAPI v1.7.2 (released on 14th May)
• P1: Let's check if we have unit test coverage for the following tests for supervisor
  • capv-e2e.[It] Hardware version upgrade creates a cluster with VM hardware versions upgraded
  • capv-e2e.[It] Label nodes with ESXi host info creates a workload cluster whose nodes have the ESXi host info
• P1: Add test "When testing ClusterClass rollouts [ClusterClass]" (supervisor) (@zhanggbj)
  • PR: [WIP] 🌱 Fill up supervisor e2e test - ClusterClass rollouts #3023
• P1: Add clusterctl upgrade tests n-2, n-1 (supervisor) (@zhanggbj)
  • CAPV 1.9=>current, CAPI 1.6=>1.7, CAPV 1.10=>current, CAPI 1.7=>1.7 (same we already have on main for govmomi)
  • PR: [WIP] 🌱 Fill up supervisor e2e test - clusterctl upgrades using ClusterClass #3024
• P1: When upgrading a workload cluster using ClusterClass with RuntimeSDK [ClusterClass] (supervisor)
  • This requires implementing a minimal Runtime Extension (let's discuss before we start implementing it, and let's do the other more straightforward sub-tasks first)
• P2: Add test "When testing the machinery for scale testing" (vcsim + supervisor)
• P2: Add clusterctl upgrade tests n-3 (supervisor)
  • Same as above just go one version more back

Other improvements:

• Use a kind cluster for clusterctl upgrade tests CAPI#7613 (@fabriziopandini)
  • PR 🌱 Use kind as a secondary management cluster for clusterctl E2E tests cluster-api#10639
  • Use kind for clusterctl upgrade tests in vsphere
• Introduce Ginkgo labels (CAPI+CAPV): CAPI#9620
  • Let's discuss details on how to do this first

Reconsider if we need it later:

• Add test: When using the autoscaler with Cluster API using ClusterClass (incl. scale from zero coverage?)

govmomi has the following additional jobs (intentionally):

• accepted: it's okay if it's only tested with govmomi:
  • capv-e2e.[It] Cluster creation with [Ignition] bootstrap [PR-Blocking] Should create a workload cluster
• accepted: not surfaced in CAPV supervisor CRDs:
  • capv-e2e.[It] Cluster creation with anti affined nodes should create a cluster with anti-affined nodes
  • capv-e2e.[It] Cluster creation with storage policy should create a cluster successfully
  • capv-e2e.[It] DHCPOverrides configuration test when Creating a cluster with DHCPOverrides configured Only configures the network with the provided nameservers
• accepted: IPAM provider not supported in supervisor mode
  • capv-e2e.[It] ClusterClass Creation using Cluster API quick-start test and IPAM Provider [ClusterClass] Should create a workload cluster

[fabriziopandini]

/area supervisor
Given that most of the improvements are for supervisor mode

[zhanggbj]

I would like to start with

• P0: Add test "When testing ClusterClass changes [ClusterClass]" (supervisor) (🌱 Fill up supervisor e2e test - ClusterClass changes #3011 )

[zhanggbj]

Also would like to pick up a similar one :-)

• P1: Add test "When testing ClusterClass rollouts [ClusterClass]" (supervisor)

[zhanggbj]

It seems this one is more about testing efforts. Just raised a PR to test.

• P1: Add clusterctl upgrade tests n-2, n-1 (supervisor)

[sbueringer]

Yeah good case is that most if it just works once added :)

A few require a bit more work