Unable to run e2e tests locally

[furkatgofurov7]

What steps did you take and what happened:
When running e2e tests locally using Makefile targets (make test-e2e or make test-e2e-run) both will fail, IF you have never run the make-docker-build-e2e target on the repo (if you have an operator image with dev tag locally, you would not see this problem, but that is not the case for everyone) which builds an operator image with dev tag.

That is because make test-e2e-run sets E2E_OPERATOR_IMAGE to http://gcr.io/k8s-staging-capi-operator/cluster-api-operator:dev, spins up a kind cluster and loads that image into cluster:

Creating cluster "capi-operator-e2e" ...
 • Ensuring node image (kindest/node:v1.27.0) 🖼  ...
 ✓ Ensuring node image (kindest/node:v1.27.0) 🖼
 • Preparing nodes 📦   ...
 ✓ Preparing nodes 📦 
 • Writing configuration 📜  ...
 ✓ Writing configuration 📜
 • Starting control-plane 🕹️  ...
 ✓ Starting control-plane 🕹️
 • Installing CNI 🔌  ...
 ✓ Installing CNI 🔌
 • Installing StorageClass 💾  ...
 ✓ Installing StorageClass 💾
  INFO: The kubeconfig file for the kind cluster is /var/folders/cz/q854zvyj34nccdhvq_4cxhd80000gp/T/e2e-kind2691374665
  INFO: Loading image: "gcr.io/k8s-staging-capi-operator/cluster-api-operator:dev"
  INFO: Image gcr.io/k8s-staging-capi-operator/cluster-api-operator:dev not present in local container image cache, will pull
  INFO: [WARNING] Unable to load image "gcr.io/k8s-staging-capi-operator/cluster-api-operator:dev" into the kind cluster "capi-operator-e2e": error pulling image "gcr.io/k8s-staging-capi-operator/cluster-api-operator:dev": failure pulling container image: Error response from daemon: manifest for gcr.io/k8s-staging-capi-operator/cluster-api-operator:dev not found: manifest unknown: Failed to fetch "dev" from request "/v2/k8s-staging-capi-operator/cluster-api-operator/manifests/dev".

Later on in the tests, operator deployment will not come up properly and fail:

state:
      waiting:
        message: Back-off pulling image "[gcr.io/k8s-staging-capi-operator/cluster-api-operator:dev](http://gcr.io/k8s-staging-capi-operator/cluster-api-operator:dev)"
        reason: ImagePullBackOff

What did you expect to happen:
run make test-e2e and make test-e2e-run successfully

To reproduce:

# in case you have run the `make-docker-build-e2e` before
$docker rmi gcr.io/k8s-staging-capi-operator/cluster-api-operator:dev
$make test-e2e-run

Tests time out waiting for capi-operator-system/capi-operator-controller-manager deployment to be available

Additional information:
I see we have 2 options in this case:

1. passing make-docker-build-e2e target to make test-e2e-run so that we always build the image first before running e2e tests locally
2. leave it to the user and document it properly somewhere mentioning that, running make-docker-build-e2e is a prerequisite for successfully running e2e tests locally

Any other suggestions?

Environment:

• Cluster-api-operator version:main
• Cluster-api version:v1.4.2
• Minikube/KIND version:1.27.0
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):macOS

/kind bug
[One or more /area label. See https://github.com/kubernetes-sigs/cluster-api-operator/labels?q=area for the list of labels]

[Fedosin]

Cluster API does nothing: https://github.com/kubernetes-sigs/cluster-api/blob/main/Makefile#L836

So, I think we just need to document this behavior and allow users to run "make docker-build-e2e && make test-e2e" if they don't have the image built.

[furkatgofurov7]

Cluster API does nothing: https://github.com/kubernetes-sigs/cluster-api/blob/main/Makefile#L836

So, I think we just need to document this behavior and allow users to run "make docker-build-e2e && make test-e2e" if they don't have the image built.

yes, was checking it as well. Something like https://github.com/kubernetes-sigs/cluster-api/blob/f3e3bda15c62f6cecb235c943f4ff337ce4ab5d1/docs/book/src/developer/testing.md?plain=1#L178C1-L180 should suffice in that case, but not sure where we need to put them

[furkatgofurov7]

/triage accepted
/kind documentation
/help
/good-first-issue

[k8s-ci-robot]

@furkatgofurov7:
This request has been marked as suitable for new contributors.

Guidelines

Please ensure that the issue body includes answers to the following questions:

• Why are we solving this issue?
• To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
• Does this issue have zero to low barrier of entry?
• How can the assignee reach out to you for help?

For more details on the requirements of such an issue, please see here and ensure that they are met.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-good-first-issue command.

In response to this:

/triage accepted
/kind documentation
/help
/good-first-issue

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[furkatgofurov7]

/remove-kind bug

[Sajiyah-Salat]

Hello I would like to take this up. Can you please let me know where should we put this text?
I think here we could add it with # symbol. So that any new user dont get confused again.