Skip to content

Change Summary v1.9

Jump to bottom
Kiran Mova edited this page May 14, 2020 · 14 revisions

Status: Released on 15-Apr-2020

Table of contents

Security Vulnerabilities Fixed

OpenEBS v1.9 includes fixes to upgrade the alpine image version used by some of the OpenEBS containers from 3.6.5 to a more recent supported version. (To check on the list of containers updated, refer to openebs/openebs#2991

Breaking Changes

OpenEBS v1.9 release has made changes to the way Jiva volume-related Kubernetes objects (Jiva Target and Controller Deployment, Jiva Target Service) are deployed. These changes are required to tighten RBAC policies for OpenEBS Service Account and to better support Jiva volumes in production. For more details refer to the Jiva Data Engine Enhancements.

OpenEBS Velero plugin container image tag format has changed to specify only OpenEBS release version. For instance:

  • OpenEBS Velero plugin v1.9 will be available at openebs/velero-plugin:1.9.0
  • OpenEBS Velero plugin earlier versions like v1.8 included the minimum velero version required like openebs/velero-plugin:1.8.0-velero_1.0.0.

OpenEBS v1.9 deprecates the OpenEBS 0.8.x version. The steps to upgrade are here.

Important Note on Jiva Volume upgrade: If you are upgrading Jiva volumes from version 1.6 and 1.7. You must use these pre-upgrade steps to check if your jiva volumes are impacted by openebs/openebs#2956. If they are, please reach out to us on OpenEBS Slack  or Kubernetes Slack #openebs channel for helping you with the upgrade.

New Capabilities

  • [cStor] Support for creating local snapshots on cStor volume in a namespace and restore/clone the cStor volume into a different namespace. (Fixed in PRs openebs/maya#1575, openebs/velero-plugin#53).

    This feature enables the users to run storage optimized CI/CD for stateful workloads. For example, a local snapshot can be taken on the volumes in production namespace. A clone of the volumes can be created under the build/test/staging namespace. The cloned volumes will not incur any additional capacity overhead as cloned volume accesses the common data via snapshot of the source volume.

    To use this feature you need to have Velero 1.3.0 or above installed. You can instruct velero to create cstor local snapshots by specifying “local” in the VolumeSnapshotLocation CR as follows:

    spec:
  provider: openebs.io/cstor-blockstore
  config:
    namespace: <OPENEBS_NAMESPACE>
    local: "true"

    For more details please refer to this doc.

  • [Upgrade] Support for upgrading multiple volumes. (Fixed in PR openebs/maya#1655).

    This feature will allow the users to upgrade all the volumes associated with an application or an application group at a time.

    The upgrade job will allow specifying a variable number of volume names as the last parameter. As an example, you can upgrade two jiva volumes with a single upgrade job by specifying the list of pv names as follows:

     spec:
    # VERIFY the value of serviceAccountName is pointing to service account
    # created within openebs namespace. Use the non-default account.
    # by running `kubectl get sa -n <openebs-namespace>`
    serviceAccountName: openebs-maya-operator
    containers:
    - name:  upgrade
      args:
      - "jiva-volume"
      # --from-version is the current version of the volume
      - "--from-version=1.6.0"
      # --to-version is the version desired upgrade version
      - "--to-version=1.9.0"
      #List of valid Jiva Peristent Volume names. 
      - "pvc-713e3bb6-afd2-11e9-8e79-42010a800065"
      - "pvc-02320288-22c8-44d4-b5a1-fc37acf2c597"

  • [Local PV] Support for pre-selecting and reserving a group of block devices using a reserved label on BlockDevice resources called openebs.io/block-device-tag. NDM has been enhanced to associate the tagged BlockDevices to only those BlockDeviceClaims that come with a matching device tag value. (Fixed in PRs openebs/ndm#397, openebs/ndm#400, openebs/maya#1647, openebs/maya#1648 ).

    This feature allows users to specify the block devices that can be only used with Local PV or with only cStor and so forth. For example, User can dedicate a set of block devices to be used for mongodb Local PVs as follows:

    • Tag a set of high performing block devices across different nodes with a unique tag value like openebs.io/block-device-tag=mongo
    • Create a new Local PV (device) StorageClass and instruct the provisioner to claim the block devices tagged with the desired tag value as follows: 
      ---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: openebs-device-mongo
  annotations:
    openebs.io/cas-type: local
    cas.openebs.io/config: |
      - name: StorageType
        value: "device"
      - name: BlockDeviceTag
        value: "Mongo"
provisioner: openebs.io/local
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete
---
    • Launch mongodb application with PVC referring to the mongodb Local PV Storage Class. This will ensure that Local PVs created for mongodb will only make use of the block devices tagged as mongodb. At the same time NDM will also ensure that these tagged block devices can't be claimed for other purposes by a different application.

Alpha Feature Updates

  • [Support for ARM images] Automated the generation of ARM containers for Local PV, cStor and corresponding Control Plane components. (In the previous releases, the generation of NDM and Jiva arm images were automated.)

  • [Support for PowerPC images] Build scripts for generating Local PV Provisioner and Jiva images for PowerPC are included in this release. (The previous releases include fixes for generating PowerPC NDM images).

  • [ZFS Local PV] Enhanced the ZFS Local PV with the following capabilities and fixes:

  • [cStor] Enhanced the cStorVolumeReplica Status to include information about snapshots that are currently available and missing on the replica. (Fixed in PRs openebs/maya#1641, openebs/maya#1639, openebs/libcstor#49, openebs/cstor#295).

    This feature can be enabled by setting the environment variable REBUILD_ESTIMATES to true on cStor pool management deployment. The default value is false. When enabled, the cStorVolumeReplic will contain available snapshot under status.snapshots and missing snapshots under status.pendingSnapshots

    status.pendingSnapshots will include snapshots, only when the cStorVolumeReplica is under rebuilding phase due to either it is a new replica or it is recovering from a node restart or a pod restart or a network loss event.

Enhancements

  • [Jiva] Enhanced the Jiva data engine to create Jiva volume-related Kubernetes objects like Jiva Target and Controller Deployment, Jiva Target Service in OpenEBS namespace.

    In earlier versions, Jiva related objects were created in PVC namespace, which required an administrator to grant full privileges to the OpenEBS service account across all namespace. Restricting the creation of the objects only under the OpenEBS namespace will help tighten RBAC policies for OpenEBS Service Account.

    In addition to moving the objects to OpenEBS namespace, a change has been made to create separate Kubernetes Deployment objects for each of the Jiva Replica. Splitting to individual replicas was required to enable users to take a faulty jiva replica from the cluster without causing restarts to the other two healthy replicas.

    When Jiva volumes are upgraded to v1.9, the objects will be automatically migrated to the openebs namespace and replica deployment object is split into multiple deployment objects.

    After the upgrade, if you need to add a new Jiva replica, you will have to create a new Kubernetes deployment with the same values as one of the healthy replicas, modify the node selector to specify the node on which new replica has to be scheduled and deploy it.

    (Fixed in PRs: openebs/maya#1636, openebs/maya#1645, openebs/maya#1649, openebs/maya#1651, openebs/maya#1646, openebs/maya#1665)

    Note that this enhancement can impact your tooling around OpenEBS Jiva Volumes. Please contact support if you need further help in understanding this change.

  • [NDM] Support for querying additional device attributes like drive type, logical block size, physical block size, and hardware sector size and save them to the BlockDevice CR. (Fixed in PRs openebs/node-disk-manager#388, openebs/node-disk-manager#375 ).

  • [cStor] Support for re-claiming the capacity on the cStor Pools, by deleting the snapshots that have been successfully backed up to a remote location. After every successful remote backup, OpenEBS will check if any of the older backup copies maintained locally can be deleted and will proceed to delete them. openebs/maya#1644, openebs/velero-plugin#57

  • [cStor] Support for configuring the chunk size for transferring the cStor snapshot to remote S3 compatible backup locations like AWS S3 or MinIO. openebs/velero-plugin#55.

    This feature can be used to set higher values for the chunk size to make the transfers faster. (Default value is determined automatically based on the size of the snapshot to be transferred).

    To configure custom chunk size, use the parameter multiPartChunkSize in VolumeSnapshotLocations as follows:

    spec:
  config:
    bucket: velero
    multiPartChunkSize: 10Mi
    provider: aws
    region: minio
    s3ForcePathStyle: "true"
    s3Url: http://minio.velero.svc:9000
  provider: openebs.io/cstor-blockstore

Major Bug Fixes

  • [cStor] Fixes an issue where the Backup of cStor volumes with 50G or higher capacity to MinIO or AWS can fail. openebs/velero-plugin#55.
  • [Jiva] Fixes an issue where Jiva replica can fail to initialize due to partially written metadata file, caused by node/pod restart. openebs/jiva#286.
  • [Jiva] Fixes an issue where Jiva used capacity metric can report incorrect value if one of the replicas is in rebuilding state. openebs/jiva#287
  • [Jiva] Fixes an issue where the error message returned by the API call was not printed into logs. openebs/jiva#289.
  • [Jiva] Fixes an issue where Jiva cleanup jobs were not scheduled due to nodes being tainted. openebs/maya#1668.
  • [NDM] Fixes an issue where labels and annotations added to BlockDevices were not retained for NDM pod is restarted. node-disk-manager#394.
  • [Local PV] Fixes an issue that prefixed the logs from Local PV provisioner with cstor. openebs/maya#1638.
  • [Local PV] Fixes a panic in Local PV provisioner, while processing a PV delete request with a hostname that is no longer present in the cluster. openebs/maya#1662.

E2E Testing Updates

  • [New Test] Verify Velero backup and restore using local snapshot. openebs/e2e-tests#211
  • [New Test] Verify scaling up/down of cStorVolumeReplicas (with CSPC). openebs/e2e-tests#217, openebs/e2e-tests#220
  • [New Test] Verify that cStor pool is marked as read-only once it reaches roThresholdLimit. openebs/e2e-tests#232
  • [New Test] Verify ZFS Local Volume editable parameters can be modified after volume is provisioned. openebs/e2e-tests#236
  • [New Test] Verify ZFS Local Volume snapshot and clone. openebs/e2e-tests#155
  • [Enhance Test] Verify that cStor local copy of snapshot is deleted after having successfully migrated to remote location. openebs/e2e-tests#211
  • [Enhance Test] Certify Backup and Restore with Velero v1.3.0. openebs/e2e-tests#211
  • [Enhance Test] Verify that cStor backup succeeds with choas(restart) injected into openebs-apiserver while backup is in progress. openebs/e2e-tests#211
  • [Enhance Test] Verify that Jiva Kubernetes objects are deployed in OpenEBS Namespace. openebs/e2e-tests#241, openebs/e2e-tests#238
  • [Verify] ZFS Local PV data consistency tests with docker runtime restarts.
  • [Verify] Bulk volume upgrade.
  • [Verify] Provisioning of cStor Volumes with waitForFirstConsumer enabled.

E2E Platform Updates

Automated end-to-end tests are executed using the GitLab runner. The following platforms were verified.

Backward Incompatibilities

This release removes the support for managing volumes that are in OpenEBS 0.8.x version or earlier. Please upgrade your volumes in versions 0.8.2 or earlier to 1.0, before upgrading to this release

For releases prior to 1.0, please refer to the respective release notes and upgrade steps.

  • From 1.0.0: None.

  • From 1.2: If the status of CSP is either in Init or PoolCreationFailed, then cstor-pool-mgmt container in pool pod will attempt to create the pool. So, when there is a need to re-create pool for the same CSP due to ephemeral storage, CSP CR related to this pool needs to be edited to set status.phase as Init. As part of reconciliation, cstor-pool-mgmt container of pool pod will attempt to recreate the pool. Refer https://github.com/openebs/maya/pull/1401 for more details.

  • From 1.3.0: None

  • From 1.4.0: None

  • From 1.5.0: Dumping cores has been disabled by default for cStor pool and NDM daemonset pods. This can be enabled by setting an ENV variable ENABLE_COREDUMP to 1. The ENV setting can be added in cStor pool deployment for dumping core for cStor pool pod and the ENV setting can be added in ndm daemonset spec for dumping core for ndm daemonset pods.

  • From 1.6.0, 1.7.0: OpenEBS v1.8 includes a critical fix (#2956) for Jiva volumes that are running in version 1.6 and 1.7. You must use these pre-upgrade steps to check if your jiva volumes are impacted. If they are, please reach out to us on [OpenEBS Slack] (https://openebs-community.slack.com/messages/openebs-users/)  or Kubernetes Slack #openebs channel for helping you with the upgrade.

Note: As part of OpenEBS upgrade or installation, maya-apiserver pod will restart if NDM blockdevice CRDs are not created before the creation of maya-apiserver. https://github.com/openebs/maya/pull/1381

Install

Prerequisite to install

  • Kubernetes 1.13+ is installed. If using Kubernetes 1.16+ then please use OpenEBS 1.4 and above.
  • Make sure that you run the below installation steps with cluster admin context. The installation will involve creating a new Service Account and assigning to OpenEBS components.
  • Make sure that iSCSI Initiator is installed on all the Kubernetes nodes.
  • Node-Disk-Manager (NDM) helps in discovering the devices attached to Kubernetes nodes, which can be used to create storage pools. If you like to exclude some of the disks from getting discovered, update the filters in NDM Config Map to exclude paths before installing OpenEBS.
  • NDM runs as a privileged pod since it needs to access the device information. Please make the necessary changes to grant access to run in privileged mode. For example, when running in RHEL/CentOS, you may need to set the security context appropriately. Refer Configuring OpenEBS with selinux=on

Install using kubectl

kubectl apply -f https://openebs.github.io/charts/openebs-operator-1.9.0.yaml

Install using helm stable charts

helm repo update
helm install --namespace openebs --name openebs stable/openebs --version 1.9.0

For more details refer to the documentation at https://docs.openebs.io/

Upgrade

The steps to upgrade are here. Kubernetes Job-based upgrades to version 1.9 is supported only from 1.0 or higher and follow a similar process like earlier releases.

  • Upgrade OpenEBS Control Plane components.
  • Upgrade Jiva PVs to 1.9, either one at a time or multiple volumes. Note that it is recommended to scale down the applications using Jiva volumes prior to upgrading.
  • Upgrade CStor Pools to 1.9 and its associated Volumes, one at a time or multiple volumes.

For upgrading from releases prior to 1.0, please refer to the respective release upgrade here.

Uninstall

The recommended steps to uninstall are:

  • delete all the OpenEBS PVCs that were created
  • delete all the SPCs (in case of cStor)
  • ensure that no volume or pool pods are pending in terminating state kubectl get pods -n <openebs namespace>
  • ensure that no openebs cStor volume custom resources are present kubectl get cvr -n <openebs namespace>
  • delete all openebs related StorageClasses.
  • delete the openebs either via helm purge or kubectl delete

Uninstalling OpenEBS doesn't automatically delete the CRDs that were created. If you would like to remove CRDs and the associated objects completely, run the following commands:

kubectl delete crd castemplates.openebs.io
kubectl delete crd cstorpools.openebs.io
kubectl delete crd cstorvolumereplicas.openebs.io
kubectl delete crd cstorvolumes.openebs.io
kubectl delete crd runtasks.openebs.io
kubectl delete crd upgradetasks.openebs.io
kubectl delete crd storagepoolclaims.openebs.io
kubectl delete crd storagepools.openebs.io
kubectl delete crd volumesnapshotdatas.volumesnapshot.external-storage.k8s.io
kubectl delete crd volumesnapshots.volumesnapshot.external-storage.k8s.io
kubectl delete crd disks.openebs.io
kubectl delete crd blockdevices.openebs.io
kubectl delete crd blockdeviceclaims.openebs.io
kubectl delete crd cstorbackups.openebs.io
kubectl delete crd cstorrestores.openebs.io
kubectl delete crd cstorcompletedbackups.openebs.io
kubectl delete crd cstorvolumeclaims.openebs.io
kubectl delete crd cstorpoolclusters.openebs.io
kubectl delete crd cstorpoolinstances.openebs.io

Note: As part of deleting the Jiva Volumes - OpenEBS launches scrub jobs for clearing data from the nodes. The completed jobs need to be cleared using the following command:

kubectl delete jobs -l openebs.io/cas-type=jiva -n <namespace>

Limitations / Known Issues

For a more comprehensive list of open issues uncovered through e2e and community testing, please refer to GitHub open issues. Here is a quick summary of common known issues.

  • The current version of OpenEBS cStor is not optimized for performance-sensitive applications.

  • Upgrade of alpha features like cStor Pools using new schema (CSPC), cStor CSI Driver, Jiva CSI Driver, MayaStor and ZFS Local PV are not supported.

  • A healthy volume(cStor or Jiva) which has a slow replica due to disk slowness or has huge network latency can cause volume read-only. In this case, a write IO to that slow replica takes more than 15-30 seconds to communicate with its controller which might cause disconnection from the initiator.

  • Over provisioning is enabled by default on cStor volumes. In case of application such as ElasticSearch, Postgresql which uses ext4 filesystem without unmap support and when a data is written and modified, ext4 tends to use new blocks of storage without sending any delete (unmap) requests back to the underlying block storage - like cStor. This can be avoided by setting a configuration on storage class based on resource quota to set a limit on the sum of capacity allocated to all the PVCs to be within the available capacity of the underlying cStor Pools. Further details are tracked here.

  • cStor volume will be offline when ReplicationFactor is not greater than 50% and then cStor volume will not come online automatically and then reconstructing data to recovered replicas. It requires manual steps to make the volume online and reconstruct the data to the replaced replicas from a healthy replica once the cStor volume is online.

  • If a pending PVC related to openebs-device StorageClass is deleted, there are chances of getting stale BDCs which ends up in consuming BDs. You have to manually delete the BDC to reclaim it.

  • In OpenShift 3.10 or above, NDM daemon set pods and NDM operators will not be upgraded if NDM daemon set's DESIRED count is not equal to the CURRENT count. This may happen if nodeSelectors have been used to deploy OpenEBS related pods OR if master/other nodes have been tainted in the k8s cluster.

  • Jiva Controller and Replica pods are stuck in Terminating state when any instability with the node or network happens and the only way to remove those containers is by using docker rm -f on the node. https://github.com/openebs/openebs/issues/2675

  • cStor Target or Pool pods can at times be stuck in a Terminating state. They will need to be manually cleaned up using kubectl delete with 0 sec grace period. Example: kubectl delete deploy -n openebs --force --grace-period=0

  • cStor pool pods can consume more memory when there is continuous load. This can cross the memory limit and cause pod evictions. It is recommended that you create cStor pools by setting the Memory limits and requests.

  • Jiva Volumes are not recommended if your use case requires snapshots and clone capabilities.

  • Jiva Replicas use a sparse file to store the data. When the application causes too many fragments (extents) to be created on the sparse file, the replica restart can cause replica to take longer time to get attached to the target. This issue was seen when there were 31K fragments created.

  • Volume Snapshots are dependent on the functionality provided by Kubernetes. The support is currently alpha. The only operations supported are:

    • Create Snapshot, Delete Snapshot and Clone from a Snapshot. Creation of the Snapshot uses a reconciliation loop, which would mean that a Create Snapshot operation will be retried on failure until the Snapshot has been successfully created. This may not be a desirable option in cases where Point in Time snapshots are expected.

  • If you are using K8s version earlier than 1.12, in certain cases, it will be observed that when the node hosting the target pod is offline, the target pod can take more than 120 seconds to get rescheduled. This is because target pods are configured with Tolerations based on the Node Condition, and TaintNodesByCondition is available only from K8s 1.12. If running an earlier version, you may have to enable the alpha gate for TaintNodesByCondition. If there is an active load on the volume when the target pod goes offline, the volume will be marked as read-only.

  • If you are using K8s version 1.13 or later, that includes the checks on ephemeral storage limits on the Pods, there is a chance that OpenEBS cStor and Jiva pods can get evicted - because there are no ephemeral requests specified. To avoid this issue, you can specify the ephemeral storage requests in the storage class or storage pool claim. (https://github.com/openebs/openebs/issues/2294)

  • When the disks used by a cStor Pool are detached and reattached, the cStor Pool may miss detecting this event in certain scenarios. Manual intervention may be required to bring the cStor Pool online. (https://github.com/openebs/openebs/issues/2363)

  • When the underlying disks used by cStor or Jiva volumes are under disk pressure due to heavy IO load, and if the Replicas take longer than 60 seconds to process the IO, the Volumes will get into Read-Only state. In 0.8.1, logs have been added to the cStor and Jiva replicas to indicate if IO has longer latency. (https://github.com/openebs/openebs/issues/2337)

  • LocalPV RawBlock volumes are not supported when the application container is running in privileged mode.

  • Upgrade of alpha features like cStor Pools using new schema (CSPC), cStor CSI Driver, MayaStor and ZFS Local PV are not supported.

  • An application may go into read-only state during the Jiva upgrade if multipath support is enabled on the Nodes. It requires manual relogin of iscsi on the node where the application pod is running or scheduling of the application onto another node to make the application into running state.

Support

Clone this wiki locally