Minio Operator PVC stuck on pending

[ajunca]

Hi,

I'm trying to deploy minio on a kubernetes cluster. I have 6 worker nodes that are reserved for minio. They are tainted (so no other pods get scheduled on this nodes). Each of this workers have two persistent volume (standard local-storage) assigned using labels, so each node has a label similar to "hds-name=hds-1..6" and each pv has a nodeAffinity pointing to his worker node (two for each node). So far so good.

The problem comes when I try to deploy minio using operator. My tenant deploy is a 6 pod with two drives each (a total of 12 drives). I have setup tolerations for the tainted nodes, but the deploy gets stuck on pending the PVC, they are never bound to the PV. Any idea/directions on how to accomplish that?

Any help will be really appreciated as I'm stuck in here.

EDIT: Now I'm receiving the following error when try to deploy the tenant (yesterday night with the same config file, I was not receiving it):

error: error validating "./minio-tenant-1.yml": error validating data: ValidationError(Tenant.spec): unknown field "tolerations" in io.min.minio.v2.Tenant.spec; if you choose to ignore these errors, turn validation off with --validate=false'

But in the docs indicates that spec.tolerations exists. I'm confused.

---

Here is an example of one pv config (remember I have two for each node):

apiVersion: v1
kind: PersistentVolume
metadata:
  labels:
    hds-storage: "true"
  name: pv-hds-1-1
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 800Gi
  hostPath:
    path: /opt/volume/hds1
    type: ""
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: hds-name
          operator: In
          values:
          - "hds-1"
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  volumeMode: Filesystem

And here the tenant config (I removed credentials):

## Secret to be used as MinIO Root Credentials
apiVersion: v1
kind: Secret
metadata:
  name: minio-tenant-1-root-creds-secret
type: Opaque
data:
  ## Access Key for MinIO Tenant, base64 encoded (echo -n 'minio' | base64)
  accesskey: XXX
  ## Secret Key for MinIO Tenant, base64 encoded (echo -n 'minio123' | base64)
  secretkey: XXX
---
## Secret to be used for MinIO Console
apiVersion: v1
kind: Secret
metadata:
  name: minio-tenant-1-console-secret
type: Opaque
data:
  ## Passphrase to encrypt jwt payload, base64 encoded (echo -n 'SECRET' | base64)
  CONSOLE_PBKDF_PASSPHRASE: XXX
  ## Salt to encrypt jwt payload, base64 encoded (echo -n 'SECRET' | base64)
  CONSOLE_PBKDF_SALT: XXX
  ## MinIO User Access Key (used for Console Login), base64 encoded (echo -n 'YOURCONSOLEACCESS' | base64)
  CONSOLE_ACCESS_KEY: XXX
  ## MinIO User Secret Key (used for Console Login), base64 encoded (echo -n 'YOURCONSOLESECRET' | base64)
  CONSOLE_SECRET_KEY: XXX
---
## MinIO Tenant Definition
apiVersion: minio.min.io/v2
kind: Tenant
metadata:
  name: minio-tenant-1
  ## Optionally pass labels to be applied to the statefulset pods
  labels:
    app: minio-tenant-1
  ## Annotations for MinIO Tenant Pods
  annotations:
    prometheus.io/path: /minio/prometheus/metrics
    prometheus.io/port: "9000"
    prometheus.io/scrape: "true"

## If a scheduler is specified here, Tenant pods will be dispatched by specified scheduler.
## If not specified, the Tenant pods will be dispatched by default scheduler.
# scheduler:
#  name: my-custom-scheduler

spec:
  ## Registry location and Tag to download MinIO Server image
  image: minio/minio:RELEASE.2021-03-01T04-20-55Z
  imagePullPolicy: IfNotPresent

  tolerations:
    - key: "node-role.miceli.tech"
      operator: "Equal"
      value: "hds"
      effect: "NoSchedule"
  
  ## Secret with credentials to be used by MinIO Tenant.
  ## Refers to the secret object created above.
  credsSecret:
    name: minio-tenant-1-root-creds-secret

  ## Specification for MinIO Pool(s) in this Tenant.
  pools:
    - servers: 6
      volumesPerServer: 2
      volumeClaimTemplate:
        metadata:
          name: data
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 800Gi
          storageClassName: local-storage
#    - servers: 4
#      volumesPerServer: 4
#      volumeClaimTemplate:
#        metadata:
#          name: data
#        spec:
#          accessModes:
#            - ReadWriteOnce
#          resources:
#            requests:
#              storage: 1Gi
  ## Mount path where PV will be mounted inside container(s).
  mountPath: /export
    ## Sub path inside Mount path where MinIO stores data.
    # subPath: /data

  ## Enable automatic Kubernetes based certificate generation and signing as explained in
  ## https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster
  requestAutoCert: true

  ## This field is used only when "requestAutoCert" is set to true. Use this field to set CommonName
  ## for the auto-generated certificate. Internal DNS name for the pod will be used if CommonName is
  ## not provided. DNS name format is *.minio.default.svc.cluster.local
  certConfig:
    commonName: ""
    organizationName: []
    dnsNames: []

  ## PodManagement policy for MinIO Tenant Pods. Can be "OrderedReady" or "Parallel"
  ## Refer https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#pod-management-policy
  ## for details.
  podManagementPolicy: Parallel

  ## Add environment variables to be set in MinIO container (https://github.com/minio/minio/tree/master/docs/config)
  # env:
  # - name: MINIO_BROWSER
  #   value: "off" # to turn-off browser
  # - name: MINIO_STORAGE_CLASS_STANDARD
  #   value: "EC:2"

  ## PriorityClassName indicates the Pod priority and hence importance of a Pod relative to other Pods.
  ## This is applied to MinIO pods only.
  ## Refer Kubernetes documentation for details https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass/
  # priorityClassName: high-priority

  ## Define configuration for Console (Graphical user interface for MinIO)
  ## Refer https://github.com/minio/console
  console:
    image: minio/console:v0.6.3
    replicas: 2
    consoleSecret:
      name: minio-tenant-1-console-secret

[ajunca]

I solved the issue using that example. The ident of tolerations is wrong. I corrected and now it works. I'm not sure why yesterday I was not receiving that error, I had tolerations as in the code pasted on the above post. Anyway, I assume that the docs are wrong? Tolerations are not under specs.tolerations, or I'm mistaking something?