-
Notifications
You must be signed in to change notification settings - Fork 929
Change Summary v1.7
- Security Vulnerabilities Fixed
- Breaking Changes
- New Capabilities
- Alpha Feature Updates
- Enhancements
- Major Bug Fixes
- E2E Testing Updates
- E2E Platform Updates
- Backward Incompatibilities
- Install
- Upgrade
- Uninstall
- Limitations / Known Issues
- Support
None
This release deprecates OpenEBS 0.6 version. The steps to upgrade are here.
None
-
[ARM] Automated the generation of ARM containers for NDM components. https://github.com/openebs/node-disk-manager/pull/371
-
[ZFS Local PV] Enhanced the ZFS Local PV with the following capabilities:
- Added support to manage snapshots and clones. https://github.com/openebs/zfs-localpv/pull/39
-
[cStor CSI volume] Enhanced the cStor CSI Driver and Pools based on the new schema with the following:
- Setup PDB on cStor Pool pods. https://github.com/openebs/maya/pull/1573
- Optimize the scheduling of cStor Replicas for Stateful sets. https://github.com/openebs/maya/pull/1591
- Improve validation of CSPC CR management. https://github.com/openebs/maya/pull/1579 https://github.com/openebs/maya/pull/1594 https://github.com/openebs/maya/pull/1598 https://github.com/openebs/maya/pull/1611
- Improve the stability of cStor CSI based snapshot operations to handle duplicate snapshot requests. https://github.com/openebs/cstor-csi/pull/70 https://github.com/openebs/cstor-csi/pull/71
- Proposal for migrating the alpha and beta schemas to v1. https://github.com/openebs/openebs/pull/2897 https://github.com/openebs/openebs/pull/2910
-
[Jiva CSI volumes] Enhance the Jiva CSI Driver and Jiva operator:
- Add support for JivaVolumePolicy CR to help with automating the Jiva volume operations. https://github.com/openebs/jiva-operator/pull/8
- Support volume resize. https://github.com/openebs/jiva-csi/pull/8
-
[NDM] Enhance the
kubectl get bd
to display the block devicepath
information. https://github.com/openebs/node-disk-manager/pull/367 -
[cStor, NDM] Refactored the NDM and cStor containers to save the core-dump files generated during error into component-specific folders like
/var/openebs/ndm
or/var/openebs/cstor
and so forth. https://github.com/openebs/cstor/pull/286 https://github.com/openebs/node-disk-manager/pull/362 -
[cStor] Added a new parameter called
thickProvisioning
to SPC. Setting it totrue
will ensure that cStor volumes can only be provisioned if the requested capacity is below the available capacity of a cStor Pool. The default value is false, which allows the overprovisioning of cStor volumes. https://github.com/openebs/maya/pull/1597 -
[Jiva] Added a new tunable to the Jiva controller to specify the grace timeout (duration) permitted for Jiva replicas to initialize their cache after a restart. For larger volumes with 300GB of data, it was noticed that the 10 sec grace period was inadequate to initialize. The new default value is 30 sec. https://github.com/openebs/jiva/pull/272
-
[Upgrade] As part of the upgrade to the latest release, remove the older unused internal CRs used to maya-apiserver. https://github.com/openebs/maya/pull/1595
-
[cStor] Fixes an issue where the metrics counter was updated with incoming IO that was responded with an error condition. https://github.com/openebs/libcstor/pull/39
-
[cStor] Fixes an issue where a cStor Pool pod could mark the cStor Volume Replica as invalid during a cStor Pool pod restart. This was caused in cases where a restart causes a race condition between a terminating pod and a newly started pod. https://github.com/openebs/maya/pull/1603
-
[cStor] Fixes an issue where cStor volume related CRs were failing to get deleted in cases where the user completely removes the underlying storage devices before deleting the volumes. https://github.com/openebs/maya/pull/1587
-
[Jiva] Fixes an issue where Jiva cleanup pods were not getting scheduled on OpenShift clusters. https://github.com/openebs/maya/pull/1600
-
[New Test] Test case for verifying cStor thickProvisioning feature. https://github.com/mayadata-io/litmus/pull/137
-
[New Test] Test case for ZFS Local PV Provisioning. https://github.com/mayadata-io/litmus/pull/135
-
[New Test] Test case for verifying scale-up of cStor Pools using CSPC schema. https://github.com/mayadata-io/litmus/pull/136
-
[New Test] Test case for Jiva CSI Driver based volume provisioning. https://github.com/mayadata-io/litmus/pull/131 https://github.com/mayadata-io/litmus/pull/123
-
[New Test] Chaos test of introducing target or replica failure while the Jiva replica is rebuilding. https://github.com/mayadata-io/litmus/pull/120
-
[New Test] Test OpenEBS components upgrades. https://github.com/mayadata-io/litmus/pull/13
Automated end-to-end tests are executed using the GitLab runner. The following platforms were verified.
This release removes the support for managing volumes that are in OpenEBS 0.6 version or earlier. Please upgrade your volumes in versions 0.8.2 or earlier to 1.0, before upgrading to this release
For releases prior to 1.0, please refer to the respective release notes and upgrade steps.
-
From 1.0.0: None.
-
From 1.2: If the status of CSP is either in
Init
orPoolCreationFailed
, thencstor-pool-mgmt
container in pool pod will attempt to create the pool. So, when there is a need to re-create pool for the same CSP due to ephemeral storage, CSP CR related to this pool needs to be edited to setstatus.phase
asInit
. As part of reconciliation,cstor-pool-mgmt
container of pool pod will attempt to recreate the pool. Refer https://github.com/openebs/maya/pull/1401 for more details. -
From 1.3.0: None
-
From 1.4.0: None
-
From 1.5.0: Dumping cores has been disabled by default for cStor pool and NDM daemonset pods. This can be enabled by setting an ENV variable
ENABLE_COREDUMP
to1
. The ENV setting can be added in cStor pool deployment for dumping core for cStor pool pod and the ENV setting can be added in ndm daemonset spec for dumping core for ndm daemonset pods. -
From 1.6.0: None
Note: As part of OpenEBS upgrade or installation, maya-apiserver
pod will restart if NDM blockdevice CRDs are not created before the creation of maya-apiserver
. https://github.com/openebs/maya/pull/1381
- Kubernetes 1.13+ is installed. If using Kubernetes 1.16+ then please use OpenEBS 1.4 and above.
- Make sure that you run the below installation steps with cluster admin context. The installation will involve creating a new Service Account and assigning to OpenEBS components.
- Make sure that iSCSI Initiator is installed on all the Kubernetes nodes.
- Node-Disk-Manager (NDM) helps in discovering the devices attached to Kubernetes nodes, which can be used to create storage pools. If you like to exclude some of the disks from getting discovered, update the filters in NDM Config Map to exclude paths before installing OpenEBS.
- NDM runs as a privileged pod since it needs to access the device information. Please make the necessary changes to grant access to run in privileged mode. For example, when running in RHEL/CentOS, you may need to set the security context appropriately. Refer Configuring OpenEBS with selinux=on
kubectl apply -f https://openebs.github.io/charts/openebs-operator-1.7.0.yaml
helm repo update
helm install --namespace openebs --name openebs stable/openebs --version 1.7.0
For more details refer to the documentation at: https://docs.openebs.io/
The steps to upgrade are here. Kubernetes Job based upgrades to version 1.7 is supported only from 1.0 or higher and follows a similar process like earlier releases.
- Upgrade OpenEBS Control Plane components.
- Upgrade Jiva PVs to 1.7, one at a time
- Upgrade CStor Pools to 1.7 and its associated Volumes, one at a time.
For upgrading from releases prior to 1.0, please refer to the respective release upgrade here.
The recommended steps to uninstall are:
- delete all the OpenEBS PVCs that were created
- delete all the SPCs (in case of cStor)
- ensure that no volume or pool pods are pending in terminating state
kubectl get pods -n <openebs namespace>
- ensure that no openebs cStor volume custom resources are present
kubectl get cvr -n <openebs namespace>
- delete all openebs related StorageClasses.
- delete the openebs either via
helm purge
orkubectl delete
Uninstalling OpenEBS doesn't automatically delete the CRDs that were created. If you would like to remove CRDs and the associated objects completely, run the following commands:
kubectl delete crd castemplates.openebs.io
kubectl delete crd cstorpools.openebs.io
kubectl delete crd cstorvolumereplicas.openebs.io
kubectl delete crd cstorvolumes.openebs.io
kubectl delete crd cstorvolumereplicas.openebs.io
kubectl delete crd runtasks.openebs.io
kubectl delete crd upgradetasks.openebs.io
kubectl delete crd storagepoolclaims.openebs.io
kubectl delete crd storagepools.openebs.io
kubectl delete crd volumesnapshotdatas.volumesnapshot.external-storage.k8s.io
kubectl delete crd volumesnapshots.volumesnapshot.external-storage.k8s.io
kubectl delete crd disks.openebs.io
kubectl delete crd blockdevices.openebs.io
kubectl delete crd blockdeviceclaims.openebs.io
kubectl delete crd cstorbackups.openebs.io
kubectl delete crd cstorrestores.openebs.io
kubectl delete crd cstorcompletedbackups.openebs.io
kubectl delete crd cstorvolumeclaims.openebs.io
kubectl delete crd cstorpoolclusters.openebs.io
kubectl delete crd cstorpoolinstances.openebs.io
Note: As part of deleting the Jiva Volumes - OpenEBS launches scrub jobs for clearing data from the nodes. The completed jobs need to be cleared using the following command:
kubectl delete jobs -l openebs.io/cas-type=jiva -n <namespace>
For a more comprehensive list of open issues uncovered through e2e and community testing, please refer to GitHub open issues. Here is a quick summary of common known issues.
-
The current version of OpenEBS cStor is not optimized for performance-sensitive applications.
-
LocalPV RawBlock volumes are not supported when the application container is running in privileged mode. Upgrade of alpha features like cStor Pools using new schema (CSPC), cStor CSI Driver, Jiva CSI Driver, MayaStor and ZFS Local PV are not supported.
-
A healthy volume(cStor or Jiva) which has a slow replica due to disk slowness or has huge network latency can cause volume read-only. In this case, a write IO to that slow replica takes more than 15-30 seconds to communicate with its controller which might cause disconnection from the initiator.
-
Over provisioning is enabled by default on cStor volumes. In case of application such as ElasticSearch, Postgresql which uses
ext4
filesystem without unmap support and when a data is written and modified,ext4
tends to use new blocks of storage without sending any delete (unmap) requests back to the underlying block storage - like cStor. This can be avoided by setting a configuration on storage class based on resource quota to set a limit on the sum of capacity allocated to all the PVCs to be within the available capacity of the underlying cStor Pools. Further details are tracked here. -
cStor volume will be offline when ReplicationFactor is not greater than 50% and then cStor volume will not come online automatically and then reconstructing data to recovered replicas. It requires manual steps to make the volume online and reconstruct the data to the replaced replicas from a healthy replica once the cStor volume is online.
-
If a pending PVC related to openebs-device StorageClass is deleted, there are chances of getting stale BDCs which ends up in consuming BDs. You have to manually delete the BDC to reclaim it.
-
In OpenShift 3.10 or above, NDM daemon set pods and NDM operators will not be upgraded if NDM daemon set's DESIRED count is not equal to the CURRENT count. This may happen if
nodeSelectors
have been used to deploy OpenEBS related pods OR if master/other nodes have been tainted in the k8s cluster. -
Jiva Controller and Replica pods are stuck in Terminating state when any instability with the node or network happens and the only way to remove those containers is by using docker rm -f on the node. https://github.com/openebs/openebs/issues/2675
-
cStor Target or Pool pods can at times be stuck in a Terminating state. They will need to be manually cleaned up using kubectl delete with 0 sec grace period. Example: kubectl delete deploy -n openebs --force --grace-period=0
-
cStor pool pods can consume more memory when there is continuous load. This can cross the memory limit and cause pod evictions. It is recommended that you create cStor pools by setting the Memory limits and requests.
-
Jiva Volumes are not recommended if your use case requires snapshots and clone capabilities.
-
Jiva Replicas use a sparse file to store the data. When the application causes too many fragments (extents) to be created on the sparse file, the replica restart can cause replica to take longer time to get attached to the target. This issue was seen when there were 31K fragments created.
-
Volume Snapshots are dependent on the functionality provided by Kubernetes. The support is currently alpha. The only operations supported are:
- Create Snapshot, Delete Snapshot and Clone from a Snapshot. Creation of the Snapshot uses a reconciliation loop, which would mean that a Create Snapshot operation will be retried on failure until the Snapshot has been successfully created. This may not be a desirable option in cases where Point in Time snapshots are expected.
-
If you are using K8s version earlier than 1.12, in certain cases, it will be observed that when the node hosting the target pod is offline, the target pod can take more than 120 seconds to get rescheduled. This is because target pods are configured with Tolerations based on the Node Condition, and TaintNodesByCondition is available only from K8s 1.12. If running an earlier version, you may have to enable the alpha gate for TaintNodesByCondition. If there is an active load on the volume when the target pod goes offline, the volume will be marked as read-only.
-
If you are using K8s version 1.13 or later, that includes the checks on ephemeral storage limits on the Pods, there is a chance that OpenEBS cStor and Jiva pods can get evicted - because there are no ephemeral requests specified. To avoid this issue, you can specify the ephemeral storage requests in the storage class or storage pool claim. (https://github.com/openebs/openebs/issues/2294)
-
When the disks used by a cStor Pool are detached and reattached, the cStor Pool may miss detecting this event in certain scenarios. Manual intervention may be required to bring the cStor Pool online. (https://github.com/openebs/openebs/issues/2363)
-
When the underlying disks used by cStor or Jiva volumes are under disk pressure due to heavy IO load, and if the Replicas take longer than 60 seconds to process the IO, the Volumes will get into Read-Only state. In 0.8.1, logs have been added to the cStor and Jiva replicas to indicate if IO has longer latency. (https://github.com/openebs/openebs/issues/2337)
-
LocalPV RawBlock volumes are not supported when the application container is running in privileged mode.
-
Upgrade of alpha features like cStor Pools using new schema (CSPC), cStor CSI Driver, MayaStor and ZFS Local PV are not supported.
-
An application may go into read-only state during Jiva upgrade if multipath support is enabled on the Nodes. It requires manual relogin of iscsi on the node where application pod is running or scheduling of the application onto another node to make the application into running state.
-
Slack community
- Already signed up? Head to our discussions at #openebs-users
- Raise an issue