Replies: 3 comments 2 replies
-
Thanks for the detailed post @jsanda, and getting that conversation started here! I understand your concerns with Ginkgo and how the The concerns you are expressing are developer ones on how a BDD framework will reduce your ability to design test code the way you want to. But you cannot neglect what it brings: better collaboration on tests from all parties involved, may they be devs or not. Also, it allows us to describe the expected behaviors prior to tests being implemented (BDD being an evolution of TDD, oriented towards functional testing). Instead of writing a doc with the tests we want to implement, we'll be able to describe them with placeholders and tie the implementation to it. If this is something we want to do (BDD), then Ginkgo is probably a solid choice as a widespread framework in the Go/k8s communities. BDD would enforce readability and also provide test outputs that are easy to follow. I would rather question the use of BDD and Ginkgo in the context of unit tests. I think it's not what BDD frameworks should be used for nor where they shine. |
Beta Was this translation helpful? Give feedback.
-
After today's discussion, it seems that Ginko may present more problems than it solves. Ultimately, I think we should go with a framework that is going to allow us to identify failing test scenarios quicker and allow tests to be run in parallel as much as possible. Having tests that maybe require a few more lines of code to setup is worth it if they can be run more easily in a development environment. I am in favor of removing Ginko for our testing here, as well as migrating our unit tests away from Ginko. If we decide that we want to peruse a BDD testing approach, then Ginko maybe comes back into play. |
Beta Was this translation helpful? Give feedback.
-
Seems like the case against BDD/Gingko was strong enough for us to move back to using the |
Beta Was this translation helpful? Give feedback.
-
Overview
There is an effort underway to address the lack of test automation in the project. There are a good number of unit tests that run as part of CI. There are only a handful of integration tests that exercise k8ssandra end to end in a running Kubernetes cluster.
The choice of testing frameworks has been a hot topic as of late, particularly in the context of integration tests. Tests are currently written with Ginkgo. There have been some discussions about using Go's testing package instead.
Background
The reaper-operator and medusa-operator projects were already a work in progress before k8ssandra was created. Those projects initially used the
testing
. Both of them are developed with [operator-sdk] which usedtesting
in early versions prior to 1.0. At or around 1.0, operator-sdk started using Ginkgo.When I upgraded reaper-operator and medusa-operator to operator-sdk 1.0, I migrated over to Ginkgo simply because that is what operator-sdk was using. I had no prior experience with Ginkgo.
When we started adding unit tests for k8ssandra, we were using Ginkgo mostly because that is what has been in use.
Global setup/tear down
Test setup for integration tests can be expensive. K8ssandra is no exception to that. Most tests will involve a Cassandra cluster. Deploying a Cassandra cluster can easily take 3 minutes.
We can have multiple tests that perform different verifications related to that Cassandra cluster. Because it an expensive operation, we do not want to deploy the Cassandra cluster for each test.
Code like the following would produce very slow tests:
Creating a Cassandra cluster for each of these tests is simply not practical. What we need is a global setup for the container function like this:
where
BeforeAll
would execute only once before all of the specs in theContext
. Ginkgo does not currently provide anything like this,Ginkgo lacks global setup/teardown for the containers,
Describe
andContext
.This Ginkgo issue has a great discussion around some of these challeneges.
It is worth noting that the Ginkgo 2.0 proposal discusses adding support for
BeforeAll
andAfterAll
.Large specs
Due to the lack of global setup/teardown functions, integration tests are often combined into one large spec.
Consider the following example:
One of the problems with large specs is the behavior of
Expect
. It fails fast. By contrast let's looks at how this might be written withtesting
:The
Error
function logs the error message and fails at the end of the test. This makes it much easier to perform multiple verifications without having to going through the test setup multiple times.Readability
One of the discussion points on which framework to use has centered readability. We want tests to be self-documenting.
Simply using one framework over the other does not make the test code inherently more readable. This is clear from the previous examples. I think it is fair to say though that tests written with Ginkgo are inherently more verbose than those written with
testing
. This is by design.Table Tests
Table-driven tests are very common in Go. Ginkgo does have support for them. Here is an example from the docs:
Here is a corresponding version using the
testing
package:On the surface the Ginkgo version may look simpler. It is actually a lot more involved. Here is what the docs say:
Test execution order
With Ginkgo I am not entirely clear on what is random by default. With
testing
tests are executed in random order. Being able to control execution order can be very helpful with integration tests. With subtests however, you have complete control over the execution order.Conclusion
This is not an exhaustive comparative analysis of Ginkgo and
testing
, nor is it intended to be. The features and differences that I have highlighted here make me believetesting
is the better choice for end-to-end integration tests.Beta Was this translation helpful? Give feedback.
All reactions