update script ci-conformance.sh #1816

prashantrewar · 2024-01-10T09:09:37Z

What this PR does / why we need it:

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #1800

Special notes for your reviewer:

Please confirm that if this PR changes any image versions, then that's the sole change this PR makes.

TODOs:

squashed commits
if necessary:
- includes documentation
- adds unit tests

/hold

k8s-ci-robot · 2024-01-10T09:09:42Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: prashantrewar
Once this PR has been reviewed and has the lgtm label, please assign fabriziopandini for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

netlify · 2024-01-10T09:09:42Z

✅ Deploy Preview for kubernetes-sigs-cluster-api-openstack ready!

Name	Link
🔨 Latest commit	`eb9c151`
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-sigs-cluster-api-openstack/deploys/65ad465a2df7800008984570
😎 Deploy Preview	https://deploy-preview-1816--kubernetes-sigs-cluster-api-openstack.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

k8s-ci-robot · 2024-01-10T09:09:46Z

Welcome @prashantrewar!

It looks like this is your first PR to kubernetes-sigs/cluster-api-provider-openstack 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/cluster-api-provider-openstack has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2024-01-10T09:09:47Z

Hi @prashantrewar. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

mdbooth · 2024-01-10T10:22:06Z

These are our prow jobs: https://github.com/kubernetes/test-infra/tree/master/config/jobs/kubernetes-sigs/cluster-api-provider-openstack

We run the conformance tests:

As a periodic: periodic-cluster-api-provider-openstack-conformance-test-main-with-k8s-ci-artifacts
As an optional presubmit: pull-cluster-api-provider-openstack-conformance
As a postsubmit: ci-cluster-api-provider-openstack-conformance

I don't understand the purpose of running this as a postsubmit, btw, although I'm still a bit unclear on the intended function of postsubmits. @lentzi90 should we delete that?

Note that in each case we invoke the test by running ./scripts/ci-conformance.sh.

./scripts/ci-conformance.sh is in our repo, and ultimately invokes make test-conformance-fast:

cluster-api-provider-openstack/scripts/ci-conformance.sh

Lines 84 to 87 in 0a4899a

 export OPENSTACK_CLOUD_YAML_FILE 

 OPENSTACK_CLOUD_YAML_FILE="$(pwd)/clouds.yaml" 

 make test-conformance-fast 

 test_status="${?}"

So if we delete test-conformance-fast we'd also have to update ci-conformance.sh.

I'm still really interested to understand how other CAPI providers handle running the conformance tests, btw. If you have time to investigate that I'd be very grateful.

/ok-to-test
/test pull-cluster-api-provider-openstack-conformance

mdbooth · 2024-01-10T11:13:02Z

/test pull-cluster-api-provider-openstack-conformance

k8s-ci-robot · 2024-01-10T11:13:05Z

@mdbooth: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test pull-cluster-api-provider-openstack-build
/test pull-cluster-api-provider-openstack-e2e-test
/test pull-cluster-api-provider-openstack-test

The following commands are available to trigger optional jobs:

/test pull-cluster-api-provider-openstack-conformance-test
/test pull-cluster-api-provider-openstack-e2e-full-test

Use /test all to run the following jobs that were automatically triggered:

pull-cluster-api-provider-openstack-build
pull-cluster-api-provider-openstack-e2e-test
pull-cluster-api-provider-openstack-test

In response to this:

/test pull-cluster-api-provider-openstack-conformance

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

mdbooth · 2024-01-10T11:13:20Z

/test pull-cluster-api-provider-openstack-conformance-test

lentzi90 · 2024-01-10T12:17:55Z

I don't understand the purpose of running this as a postsubmit, btw, although I'm still a bit unclear on the intended function of postsubmits. @lentzi90 should we delete that?

Yeah I'm not sure why we have it as both postsubmit and periodic... I think we could delete the postsubmit.

I'm still really interested to understand how other CAPI providers handle running the conformance tests, btw. If you have time to investigate that I'd be very grateful.

I looked at both CAPA and CAPZ. They seem to have a very similar conformance suite to what we have.

There is ci-conformance.sh
calling make test-conformance (no fast option)
the conformance check is to run kubetest
which is the same that we do

I'm not sure what our fast conformance was supposed to do but I think it is safe to delete it. CAPZ has a fast option that increases the number of ginkgo nodes and can take a different config. But this is not used in their CI.

lentzi90 · 2024-01-10T12:21:23Z

Ah now we are getting somewhere!

 You're using deprecated Ginkgo functionality:
=============================================
  Measure is deprecated and has been removed from Ginkgo V2.  Any Measure tests in your spec will not run.  Please migrate to gomega/gmeasure.
  Learn more at: https://onsi.github.io/ginkgo/MIGRATING_TO_V2#removed-measure
    /home/prow/go/src/sigs.k8s.io/cluster-api-provider-openstack/test/e2e/suites/conformance/conformance_test.go:54

@prashantrewar can you check how CAPA is doing this? They have a different way of doing this "measuring" and we should probably do the same.

prashantrewar · 2024-01-10T19:31:13Z

@lentzi90 and @mdbooth, when I replace the whole code of conformance_test.go with CAPA conformance_test.go code after that when I run make test-conformance I got an error -- A BeforeSuite node failed so all tests were skipped.

Running Suite: capo-conformance - /home/praveen/Workspace/kubernetes-sigs/cluster-api-provider-openstack/test/e2e/suites/conformance
====================================================================================================================================
Random Seed: 1704913903

Will run 1 of 1 specs
------------------------------
[SynchronizedBeforeSuite] 
/home/praveen/Workspace/kubernetes-sigs/cluster-api-provider-openstack/test/e2e/suites/conformance/conformance_suite_test.go:48
  > Enter [SynchronizedBeforeSuite] TOP-LEVEL @ 01/11/24 00:42:16.352
  [2024-01-11T00:42:16+05:30] Running Node1BeforeSuite
  [2024-01-11T00:42:16+05:30] Loading the e2e test configuration from "/home/praveen/Workspace/kubernetes-sigs/cluster-api-provider-openstack/test/e2e/data/e2e_conf.yaml"
  [2024-01-11T00:42:16+05:30] Finished Node1BeforeSuite
  [FAILED] in [SynchronizedBeforeSuite] - /home/praveen/Workspace/kubernetes-sigs/cluster-api-provider-openstack/test/e2e/shared/suite.go:66 @ 01/11/24 00:42:16.356
  < Exit [SynchronizedBeforeSuite] TOP-LEVEL @ 01/11/24 00:42:16.356 (4ms)
[SynchronizedBeforeSuite] [FAILED] [0.004 seconds]
[SynchronizedBeforeSuite] 
/home/praveen/Workspace/kubernetes-sigs/cluster-api-provider-openstack/test/e2e/suites/conformance/conformance_suite_test.go:48

  [FAILED] Invalid test suite argument. Value of environment variable OPENSTACK_CLOUD_YAML_FILE should be an existing file: ../../../../clouds.yaml
  Expected
      <string>: ../../../../clouds.yaml
  to exist
  In [SynchronizedBeforeSuite] at: /home/praveen/Workspace/kubernetes-sigs/cluster-api-provider-openstack/test/e2e/shared/suite.go:66 @ 01/11/24 00:42:16.356

  Full Stack Trace
    sigs.k8s.io/cluster-api-provider-openstack/test/e2e/shared.Node1BeforeSuite(0xc000173080)
        /home/praveen/Workspace/kubernetes-sigs/cluster-api-provider-openstack/test/e2e/shared/suite.go:66 +0x536
    sigs.k8s.io/cluster-api-provider-openstack/test/e2e/suites/conformance.glob..func1()
        /home/praveen/Workspace/kubernetes-sigs/cluster-api-provider-openstack/test/e2e/suites/conformance/conformance_suite_test.go:49 +0x1a
    reflect.Value.call({0x20bcd60?, 0x2731d38?, 0x13?}, {0x25ef136, 0x4}, {0xc000090f20, 0x0, 0x0?})
        /usr/local/go/src/reflect/value.go:596 +0xce7
    reflect.Value.Call({0x20bcd60?, 0x2731d38?, 0x0?}, {0xc000081f20?, 0x0?, 0x0?})
        /usr/local/go/src/reflect/value.go:380 +0xb9
------------------------------
[SynchronizedAfterSuite] 
/home/praveen/Workspace/kubernetes-sigs/cluster-api-provider-openstack/test/e2e/suites/conformance/conformance_suite_test.go:54
  > Enter [SynchronizedAfterSuite] TOP-LEVEL @ 01/11/24 00:42:16.356
  [2024-01-11T00:42:16+05:30] Running AllNodesAfterSuite
  bootstrap cluster proxy does not exist yet, cannot dump clusters and machines
  [2024-01-11T00:42:16+05:30] Finished AllNodesAfterSuite
  < Exit [SynchronizedAfterSuite] TOP-LEVEL @ 01/11/24 00:42:16.356 (0s)
  > Enter [SynchronizedAfterSuite] TOP-LEVEL @ 01/11/24 00:42:16.356
  [2024-01-11T00:42:16+05:30] Running Node1AfterSuite
  [2024-01-11T00:42:16+05:30] Tearing down the management cluster
  [2024-01-11T00:42:16+05:30] Finished Node1AfterSuite
  < Exit [SynchronizedAfterSuite] TOP-LEVEL @ 01/11/24 00:42:16.356 (0s)
[SynchronizedAfterSuite] PASSED [0.000 seconds]
------------------------------

Summarizing 1 Failure:
  [FAIL] [SynchronizedBeforeSuite] 
  /home/praveen/Workspace/kubernetes-sigs/cluster-api-provider-openstack/test/e2e/shared/suite.go:66

Ran 0 of 1 Specs in 0.005 seconds
FAIL! -- A BeforeSuite node failed so all tests were skipped.
--- FAIL: TestE2EConformance (0.01s)
FAIL

Ginkgo ran 1 suite in 32.994542623s

Test Suite Failed
You're using deprecated Ginkgo functionality:
=============================================
  --stream is deprecated
  Learn more at: https://onsi.github.io/ginkgo/MIGRATING_TO_V2#removed--stream

To silence deprecations that can be silenced set the following environment variable:
  ACK_GINKGO_DEPRECATIONS=2.11.0


real    0m32.526s
user    0m11.226s
sys     0m2.119s
make: *** [Makefile:203: test-conformance] Error 1

lentzi90 · 2024-01-15T07:18:58Z

Alright, instead of copy/pasting it, can you check the ginkgo docs about the deprecation and try to follow their migration suggestion. The docs are here: https://onsi.github.io/ginkgo/MIGRATING_TO_V2#removed-measure
If it doesn't work immediately, don't worry. Just push it here and we can try to figure it out together

mdbooth · 2024-01-15T11:09:56Z

FYI I've submitted a PR to remove the postsubmit here: kubernetes/test-infra#31625

Signed-off-by: Prashant Rewar <[email protected]>

k8s-ci-robot · 2024-01-21T16:41:14Z

@prashantrewar: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-cluster-api-provider-openstack-test	`eb9c151`	link	true	`/test pull-cluster-api-provider-openstack-test`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

k8s-ci-robot · 2024-04-13T00:51:48Z

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 10, 2024

k8s-ci-robot requested review from lentzi90 and mdbooth January 10, 2024 09:09

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jan 10, 2024

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jan 10, 2024

k8s-ci-robot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Jan 10, 2024

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jan 10, 2024

prashantrewar force-pushed the update-scripts branch from 88d3a17 to fb4c655 Compare January 21, 2024 13:16

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jan 21, 2024

prashantrewar force-pushed the update-scripts branch 2 times, most recently from 245401e to cf97c4d Compare January 21, 2024 13:57

update script ci-conformance.sh

eb9c151

Signed-off-by: Prashant Rewar <[email protected]>

prashantrewar force-pushed the update-scripts branch from cf97c4d to eb9c151 Compare January 21, 2024 16:29

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

update script ci-conformance.sh #1816

update script ci-conformance.sh #1816

prashantrewar commented Jan 10, 2024

k8s-ci-robot commented Jan 10, 2024

netlify bot commented Jan 10, 2024 •

edited

k8s-ci-robot commented Jan 10, 2024

k8s-ci-robot commented Jan 10, 2024

mdbooth commented Jan 10, 2024

mdbooth commented Jan 10, 2024

k8s-ci-robot commented Jan 10, 2024

mdbooth commented Jan 10, 2024

lentzi90 commented Jan 10, 2024

lentzi90 commented Jan 10, 2024

prashantrewar commented Jan 10, 2024

lentzi90 commented Jan 15, 2024

mdbooth commented Jan 15, 2024

k8s-ci-robot commented Jan 21, 2024 •

edited

k8s-ci-robot commented Apr 13, 2024

update script ci-conformance.sh #1816

Are you sure you want to change the base?

update script ci-conformance.sh #1816

Conversation

prashantrewar commented Jan 10, 2024

k8s-ci-robot commented Jan 10, 2024

netlify bot commented Jan 10, 2024 • edited

✅ Deploy Preview for kubernetes-sigs-cluster-api-openstack ready!

k8s-ci-robot commented Jan 10, 2024

k8s-ci-robot commented Jan 10, 2024

mdbooth commented Jan 10, 2024

mdbooth commented Jan 10, 2024

k8s-ci-robot commented Jan 10, 2024

mdbooth commented Jan 10, 2024

lentzi90 commented Jan 10, 2024

lentzi90 commented Jan 10, 2024

prashantrewar commented Jan 10, 2024

lentzi90 commented Jan 15, 2024

mdbooth commented Jan 15, 2024

k8s-ci-robot commented Jan 21, 2024 • edited

k8s-ci-robot commented Apr 13, 2024

netlify bot commented Jan 10, 2024 •

edited

k8s-ci-robot commented Jan 21, 2024 •

edited