[loki-distributed] Error Ingester read-only file system to S3 with TSDB in EKS

[northonheld]

Hello, even following the guidelines on how to configure S3 in the official documentation, Ingester is unable to communicate with the s3 bucket.

Chart version loki-distributed: 0.79.0

I used this doc link for these values:

  schemaConfig:
    configs:
      - from: "2020-07-01"
        store: tsdb
        object_store: aws
        schema: v13
        index:
          prefix: index_
          period: 24h

  storageConfig:
    tsdb_shipper:
     active_index_directory: /loki/index
     cache_location: /loki/index_cache
     cache_ttl: 24h
    aws:
      s3: s3://us-east-1
      bucketnames: xxxx-infra-loki

serviceAccount:
  create: true
  annotations:
    eks.amazonaws.com/role-arn: "arn:aws:iam::1XXXXXXXXX0:role/sa-bucket-loki-infra"                                                 
  automountServiceAccountToken: true

ingester:
  persistence:
    enabled: false

I created the objects in AWS using Crossplane, based on the Terraform module officially provided at this link

---
apiVersion: iam.aws.crossplane.io/v1beta1
kind: Policy
metadata:
  name: sa-bucket-loki-infra-policy
  namespace: loki
spec:
  deletionPolicy: Delete
  forProvider:
    name: sa-bucket-loki-infra-policy
    description: Allow access to S3 loki
    document: >
      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Action": [
              "s3:*"
            ],
            "Resource": [
              "arn:aws:s3:::xxxx-infra-loki",
              "arn:aws:s3:::xxxx-infra-loki/*"
            ]
          }
        ]
      }
  providerConfigRef:
    name: crossplane-cred-infra
---
apiVersion: iam.aws.crossplane.io/v1beta1
kind: Role
metadata:
  name: sa-bucket-loki-infra-role
  namespace: loki
spec:
  forProvider:
    assumeRolePolicyDocument: |
      {
          "Version": "2012-10-17",
          "Statement": [
              {
                  "Effect": "Allow",
                  "Action": "sts:AssumeRoleWithWebIdentity",
                  "Principal": {
                      "Federated": "arn:aws:iam::1XXXXXXXXXX0:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/4XXXXXXXXXXXXXXXXXXXXXXXXXXXXX8"
                  },
                  "Condition": {
                      "StringEquals": {
                          "oidc.eks.us-east-1.amazonaws.com/id/4XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX8:aud": [
                              "sts.amazonaws.com"
                          ],
                          "oidc.eks.us-east-1.amazonaws.com/id/4XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX8:sub": [
                              "system:serviceaccount:loki:loki-eks-infra-loki-distributed"
                          ]
                      }
                  }
              }
          ]
      }
    tags:
      - key: loki
        value: infra
  providerConfigRef:
    name: crossplane-cred-infra
---
apiVersion: iam.aws.crossplane.io/v1beta1
kind: RolePolicyAttachment
metadata:
  name: sa-bucket-loki-infra-att
  namespace: loki
spec:
  forProvider:
    policyArnRef:
      name: sa-bucket-loki-infra-policy
    roleNameRef:
      name: sa-bucket-loki-infra-role
  providerConfigRef:
    name: crossplane-cred-infra
---
apiVersion: s3.aws.crossplane.io/v1beta1
kind: Bucket
metadata:
  name: xxxxx-infra-loki
  namespace: loki
spec:
  deletionPolicy: Orphan
  forProvider:
    locationConstraint: us-east-1
    objectOwnership: BucketOwnerEnforced
    paymentConfiguration:
      payer: BucketOwner
    publicAccessBlockConfiguration:
      blockPublicAcls: true
      blockPublicPolicy: true
      ignorePublicAcls: true
      restrictPublicBuckets: true
    tagging:
      tagSet:
        - key: xxxx
          value: xxx
  providerConfigRef:
    name: crossplane-cred-infra

I think created it correctly:
Trust relationsships from role:

Policy Attachment:

I checked the annotation of the service account in eks and the annotation is correct:
kubectl get sa/loki-eks-infra-loki-distributed -n loki -o yaml

apiVersion: v1
automountServiceAccountToken: true
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::1XXXXXXXXX0:role/sa-bucket-loki-infra-role
  labels:
    app.kubernetes.io/instance: loki-eks-infra
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: loki-distributed
    app.kubernetes.io/version: 2.9.6
    argocd.argoproj.io/instance: loki-eks-infra
    helm.sh/chart: loki-distributed-0.79.0
  name: loki-eks-infra-loki-distributed
  namespace: loki

And Statefulset ingester:
kubectl get sts/loki-eks-infra-loki-distributed-ingester -n loki -o yaml

      serviceAccount: loki-eks-infra-loki-distributed
      serviceAccountName: loki-eks-infra-loki-distributed

But I'm still getting this error in Ingester:

loki-eks-infra-loki-distributed-ingester-0 ingester mkdir /loki/index: read-only file system
loki-eks-infra-loki-distributed-ingester-0 ingester error initialising module: store
loki-eks-infra-loki-distributed-ingester-0 ingester github.com/grafana/dskit/modules.(*Manager).initModule
loki-eks-infra-loki-distributed-ingester-0 ingester 	/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:138
loki-eks-infra-loki-distributed-ingester-0 ingester github.com/grafana/dskit/modules.(*Manager).InitModuleServices
loki-eks-infra-loki-distributed-ingester-0 ingester 	/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108
loki-eks-infra-loki-distributed-ingester-0 ingester github.com/grafana/loki/pkg/loki.(*Loki).Run
loki-eks-infra-loki-distributed-ingester-0 ingester 	/src/loki/pkg/loki/loki.go:461
loki-eks-infra-loki-distributed-ingester-0 ingester main.main
loki-eks-infra-loki-distributed-ingester-0 ingester 	/src/loki/cmd/loki/main.go:110
loki-eks-infra-loki-distributed-ingester-0 ingester runtime.main
loki-eks-infra-loki-distributed-ingester-0 ingester 	/usr/local/go/src/runtime/proc.go:267
loki-eks-infra-loki-distributed-ingester-0 ingester runtime.goexit
loki-eks-infra-loki-distributed-ingester-0 ingester 	/usr/local/go/src/runtime/asm_amd64.s:1650

I can't figure out where my mistake lies. I followed all the guidelines in the official documentation. Why doesn't it work? Did I forget to enable something?

This is the current view of my pods

NAME                                                              READY   STATUS             RESTARTS          AGE
loki-eks-infra-loki-distributed-distributor-6955bf69b7-4p76p      1/1     Running            0                 66m
loki-eks-infra-loki-distributed-distributor-6955bf69b7-n6zz9      1/1     Running            0                 3h8m
loki-eks-infra-loki-distributed-index-gateway-0                   0/1     CrashLoopBackOff   506 (64s ago)     42h
loki-eks-infra-loki-distributed-ingester-0                        0/1     CrashLoopBackOff   21 (3m27s ago)    86m
loki-eks-infra-loki-distributed-ingester-1                        0/1     CrashLoopBackOff   21 (3m21s ago)    86m
loki-eks-infra-loki-distributed-ingester-2                        0/1     CrashLoopBackOff   21 (3m33s ago)    86m
loki-eks-infra-loki-distributed-querier-769d9f7c75-dfr2v          1/1     Running            0                 66m
loki-eks-infra-loki-distributed-querier-769d9f7c75-s8zzj          1/1     Running            0                 3h9m
loki-eks-infra-loki-distributed-query-frontend-6c8f5f5f54-cbh46   0/1     Running            9 (3m13s ago)     51m
loki-eks-infra-loki-distributed-query-frontend-c576b64-gfn4p      0/1     Running            481 (5m16s ago)   42h