Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Conformance tests do not run any tests #1800

Open
lentzi90 opened this issue Dec 21, 2023 · 8 comments · May be fixed by #1816
Open

Conformance tests do not run any tests #1800

lentzi90 opened this issue Dec 21, 2023 · 8 comments · May be fixed by #1816
Assignees
@prashantrewar
Labels
good first issue Denotes an issue ready for a new contributor, according to the "help wanted" guidelines. help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. kind/bug Categorizes issue or PR as related to a bug.

Comments

@lentzi90
Copy link
Contributor

/kind bug

What steps did you take and what happened:

As can be seen in the logs, the conformance tests do not run any tests as it is now.

 Will run 0 of 0 specs
Ran 0 of 0 Specs in 0.000 seconds
SUCCESS! -- 0 Passed | 0 Failed | 0 Pending | 0 Skipped
PASS

It should definitely run some tests!

What did you expect to happen:

These tests should run.

Anything else you would like to add:

I think this is just a question of selecting tests correctly in the config or adjusting the make target or script.

/help
/good-first-issue
@k8s-ci-robot
Copy link
Contributor

@lentzi90:
This request has been marked as suitable for new contributors.

Guidelines

Please ensure that the issue body includes answers to the following questions:

  • Why are we solving this issue?
  • To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
  • Does this issue have zero to low barrier of entry?
  • How can the assignee reach out to you for help?

For more details on the requirements of such an issue, please see here and ensure that they are met.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-good-first-issue command.

In response to this:

/kind bug

What steps did you take and what happened:

As can be seen in the logs, the conformance tests do not run any tests as it is now.

Will run 0 of 0 specs
Ran 0 of 0 Specs in 0.000 seconds
SUCCESS! -- 0 Passed | 0 Failed | 0 Pending | 0 Skipped
PASS

It should definitely run some tests!

What did you expect to happen:

These tests should run.

Anything else you would like to add:

I think this is just a question of selecting tests correctly in the config or adjusting the make target or script.

/help
/good-first-issue

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added kind/bug Categorizes issue or PR as related to a bug. good first issue Denotes an issue ready for a new contributor, according to the "help wanted" guidelines. help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. labels Dec 21, 2023
@prashantrewar
Copy link

hey @lentzi90 , i would like to work on this, could you please guide me for this issue.

@lentzi90
Copy link
Contributor Author

lentzi90 commented Jan 8, 2024

Sure!
I think the issue is that ginkgo is skipping the tests. I'm not sure exactly why, but this is how it works:

  1. The prow job is defined here: https://github.com/kubernetes/test-infra/blob/73538ad97a1f454e05e6c31fcd74bdd7ac89efaf/config/jobs/kubernetes-sigs/cluster-api-provider-openstack/cluster-api-provider-openstack-periodics.yaml#L52-L94
    For other jobs we set some environment variables like here that could affect what tests are run. Maybe we need to add something?
  2. The job runs this script: https://github.com/kubernetes-sigs/cluster-api-provider-openstack/blob/main/scripts/ci-conformance.sh
  3. The script executes make test-conformance-fast:

    cluster-api-provider-openstack/Makefile

    Lines 205 to 206 in fdc5f49

    test-conformance-fast: ## Run clusterctl based conformance test on workload cluster (requires Docker) using a subset of the conformance suite in parallel.
    $(MAKE) test-conformance CONFORMANCE_E2E_ARGS="-kubetest.config-file=$(KUBETEST_FAST_CONF_PATH) -kubetest.ginkgo-nodes=5 $(E2E_ARGS)"

    This is where I suspect we have some issue. I think we need to add a -focus=conformance perhaps.

Is there anything in particular you are wondering?

@lentzi90
Copy link
Contributor Author

lentzi90 commented Jan 8, 2024

@mdbooth any idea what the difference between conformance and conformance-fast is supposed to be?

cluster-api-provider-openstack/Makefile

Lines 202 to 206 in fdc5f49

test-conformance: e2e-prerequisites ## Run clusterctl based conformance test on workload cluster (requires Docker).
time $(GINKGO) -trace -show-node-events -v -tags=e2e -focus="conformance" $(CONFORMANCE_GINKGO_ARGS) ./test/e2e/suites/conformance/... -- -config-path="$(E2E_CONF_PATH)" -artifacts-folder="$(ARTIFACTS)" --data-folder="$(E2E_DATA_DIR)" $(CONFORMANCE_E2E_ARGS)
test-conformance-fast: ## Run clusterctl based conformance test on workload cluster (requires Docker) using a subset of the conformance suite in parallel.
$(MAKE) test-conformance CONFORMANCE_E2E_ARGS="-kubetest.config-file=$(KUBETEST_FAST_CONF_PATH) -kubetest.ginkgo-nodes=5 $(E2E_ARGS)"

I'm getting the feeling that the "fast" test is not meant to do much, but is it supposed to skip all the tests? 🤔

@prashantrewar
Copy link

/assign

@mdbooth
Copy link
Contributor

mdbooth commented Jan 9, 2024

@mdbooth any idea what the difference between conformance and conformance-fast is supposed to be?

I'm getting the feeling that the "fast" test is not meant to do much, but is it supposed to skip all the tests? 🤔

I don't, I'm afraid: I've never been down this rabbit hole. However, I agree that skipping all the tests is clearly not useful. I'm guessing this is a copied pattern, though. Do other providers have these targets, and if so what do they do? I normally look at CAPA and CAPZ first for this kind of thing.

@lentzi90
Copy link
Contributor Author

Good idea!
CAPA has no fast conformance variant, CAPZ has, but it seems to be just higher parallelism and a possibility to use special config. From what I can tell both use the normal conformance test in CI. I suggest we try the same.

@prashantrewar could you change the script to do make test-conformance instead of make test-conformance-fast, and remove the test-conformance-fast target all together?

@prashantrewar prashantrewar linked a pull request Jan 10, 2024 that will close this issue
Open
3 tasks
@prashantrewar
Copy link

Hey @lentzi90 I created a PR on this issue, could you please take a look

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@prashantrewar prashantrewar

Labels
good first issue Denotes an issue ready for a new contributor, according to the "help wanted" guidelines. help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

update script ci-conformance.sh prashantrewar/cluster-api-provider-openstack
4 participants
@mdbooth @lentzi90 @k8s-ci-robot @prashantrewar