Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Can't install cold storage on 0.32.2 #370

Open
luna215 opened this issue Jan 4, 2024 · 3 comments
Open

Can't install cold storage on 0.32.2 #370

luna215 opened this issue Jan 4, 2024 · 3 comments

Comments

@luna215
Copy link

luna215 commented Jan 4, 2024

When I run this comamand to install signoz 0.32.2 via helm:

helm --namespace platform install signoz-new signoz/signoz -f install.yaml --debug
with the following yaml file to override some values:

global:
  storageClass: gp2-resizable
  cloud: aws

clickhouse:
  installCustomStorageClass: true
  coldStorage:
    enabled: true
    defaultKeepFreeSpaceBytes: "10485760"
    endpoint: https://*******.s3.us-west-1.amazonaws.com/data/
    accessKey: ********
    secretAccess: **********
    role:
      enabled: true
      annotations:
        eks.amazonaws.com/role-arn: arn:aws:iam::***:role/***

I get the following error:

install.go:214: [debug] Original chart version: ""
install.go:231: [debug] CHART PATH: /Users/paulluna/Library/Caches/helm/repository/signoz-0.32.2.tgz

coalesce.go:237: warning: skipped value for zookeeper.initContainers: Not a table.
client.go:142: [debug] creating 1 resource(s)
install.go:168: [debug] CRD clickhouseinstallations.clickhouse.altinity.com is already present. Skipping.
client.go:142: [debug] creating 1 resource(s)
install.go:168: [debug] CRD clickhouseinstallationtemplates.clickhouse.altinity.com is already present. Skipping.
client.go:142: [debug] creating 1 resource(s)
install.go:168: [debug] CRD clickhouseoperatorconfigurations.clickhouse.altinity.com is already present. Skipping.
client.go:393: [debug] checking 55 resources for changes
client.go:414: [debug] Created a new ServiceAccount called "signoz-new-clickhouse" in platform

client.go:414: [debug] Created a new ServiceAccount called "signoz-new-clickhouse-operator" in platform

client.go:414: [debug] Created a new ServiceAccount called "signoz-new-k8s-infra-otel-agent" in platform

client.go:414: [debug] Created a new ServiceAccount called "signoz-new-k8s-infra-otel-deployment" in platform

client.go:414: [debug] Created a new ServiceAccount called "signoz-new-alertmanager" in platform

client.go:414: [debug] Created a new ServiceAccount called "signoz-new-frontend" in platform

client.go:414: [debug] Created a new ServiceAccount called "signoz-new-otel-collector-metrics" in platform

client.go:414: [debug] Created a new ServiceAccount called "signoz-new-otel-collector" in platform

client.go:414: [debug] Created a new ServiceAccount called "signoz-new-query-service" in platform

client.go:414: [debug] Created a new Secret called "signoz-new-clickhouse-operator" in platform

client.go:414: [debug] Created a new ConfigMap called "signoz-new-zookeeper-scripts" in platform

client.go:414: [debug] Created a new ConfigMap called "signoz-new-clickhouse-custom-functions" in platform

client.go:414: [debug] Created a new ConfigMap called "signoz-new-clickhouse-operator-etc-confd-files" in platform

client.go:414: [debug] Created a new ConfigMap called "signoz-new-clickhouse-operator-etc-configd-files" in platform

client.go:414: [debug] Created a new ConfigMap called "signoz-new-clickhouse-operator-etc-files" in platform

client.go:414: [debug] Created a new ConfigMap called "signoz-new-clickhouse-operator-etc-templatesd-files" in platform

client.go:414: [debug] Created a new ConfigMap called "signoz-new-clickhouse-operator-etc-usersd-files" in platform

client.go:414: [debug] Created a new ConfigMap called "signoz-new-k8s-infra-otel-agent" in platform

client.go:414: [debug] Created a new ConfigMap called "signoz-new-k8s-infra-otel-deployment" in platform

client.go:414: [debug] Created a new ConfigMap called "signoz-new-frontend" in platform

client.go:414: [debug] Created a new ConfigMap called "signoz-new-otel-collector-metrics" in platform

client.go:414: [debug] Created a new ConfigMap called "signoz-new-otel-collector" in platform

client.go:414: [debug] Created a new ConfigMap called "signoz-new-query-service" in platform

client.go:684: [debug] Looks like there are no changes for StorageClass "gp2-resizable"
client.go:684: [debug] Looks like there are no changes for ClusterRole "signoz-new-k8s-infra-otel-agent-platform"
client.go:684: [debug] Looks like there are no changes for ClusterRole "signoz-new-k8s-infra-otel-deployment-platform"
client.go:684: [debug] Looks like there are no changes for ClusterRole "signoz-new-otel-collector-metrics-platform"
client.go:684: [debug] Looks like there are no changes for ClusterRole "signoz-new-otel-collector-platform"
client.go:684: [debug] Looks like there are no changes for ClusterRoleBinding "signoz-new-k8s-infra-otel-agent-platform"
client.go:684: [debug] Looks like there are no changes for ClusterRoleBinding "signoz-new-k8s-infra-otel-deployment-platform"
client.go:684: [debug] Looks like there are no changes for ClusterRoleBinding "signoz-new-otel-collector-metrics-platform"
client.go:684: [debug] Looks like there are no changes for ClusterRoleBinding "signoz-new-otel-collector-platform"
client.go:414: [debug] Created a new Role called "signoz-new-clickhouse-operator" in platform

client.go:414: [debug] Created a new RoleBinding called "signoz-new-clickhouse-operator" in platform

client.go:414: [debug] Created a new Service called "signoz-new-zookeeper-headless" in platform

client.go:414: [debug] Created a new Service called "signoz-new-zookeeper" in platform

client.go:414: [debug] Created a new Service called "signoz-new-clickhouse-operator-metrics" in platform

client.go:414: [debug] Created a new Service called "signoz-new-k8s-infra-otel-agent" in platform

client.go:414: [debug] Created a new Service called "signoz-new-k8s-infra-otel-deployment" in platform

client.go:414: [debug] Created a new Service called "signoz-new-alertmanager" in platform

client.go:414: [debug] Created a new Service called "signoz-new-alertmanager-headless" in platform

client.go:414: [debug] Created a new Service called "signoz-new-frontend" in platform

client.go:414: [debug] Created a new Service called "signoz-new-otel-collector-metrics" in platform

client.go:414: [debug] Created a new Service called "signoz-new-otel-collector" in platform

client.go:414: [debug] Created a new Service called "signoz-new-query-service" in platform

client.go:414: [debug] Created a new DaemonSet called "signoz-new-k8s-infra-otel-agent" in platform

client.go:414: [debug] Created a new Deployment called "signoz-new-clickhouse-operator" in platform

client.go:414: [debug] Created a new Deployment called "signoz-new-k8s-infra-otel-deployment" in platform

client.go:414: [debug] Created a new Deployment called "signoz-new-frontend" in platform

client.go:414: [debug] Created a new Deployment called "signoz-new-otel-collector-metrics" in platform

client.go:414: [debug] Created a new Deployment called "signoz-new-otel-collector" in platform

client.go:414: [debug] Created a new StatefulSet called "signoz-new-zookeeper" in platform

client.go:414: [debug] Created a new StatefulSet called "signoz-new-alertmanager" in platform

client.go:414: [debug] Created a new StatefulSet called "signoz-new-query-service" in platform

client.go:414: [debug] Created a new ClickHouseInstallation called "signoz-new-clickhouse" in platform

client.go:486: [debug] Starting delete for "signoz-new-schema-migrator" Job
client.go:490: [debug] Ignoring delete failure for "signoz-new-schema-migrator" batch/v1, Kind=Job: jobs.batch "signoz-new-schema-migrator" not found
client.go:142: [debug] creating 1 resource(s)
client.go:712: [debug] Watching for changes to Job signoz-new-schema-migrator with timeout of 5m0s
client.go:740: [debug] Add/Modify event for signoz-new-schema-migrator: ADDED
client.go:779: [debug] signoz-new-schema-migrator: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
Error: INSTALLATION FAILED: failed post-install: 1 error occurred:
        * timed out waiting for the condition


helm.go:84: [debug] failed post-install: 1 error occurred:
        * timed out waiting for the condition

INSTALLATION FAILED
main.newInstallCmd.func2
        helm.sh/helm/v3/cmd/helm/install.go:154
github.com/spf13/cobra.(*Command).execute
        github.com/spf13/[email protected]/command.go:940
github.com/spf13/cobra.(*Command).ExecuteC
        github.com/spf13/[email protected]/command.go:1068
github.com/spf13/cobra.(*Command).Execute
        github.com/spf13/[email protected]/command.go:992
main.main
        helm.sh/helm/v3/cmd/helm/helm.go:83
runtime.main
        runtime/proc.go:267
runtime.goexit
        runtime/asm_arm64.s:1197

Any ideas of what can be causing the installation to error out?
@srikanthccv
Copy link
Member

@luna215 Does it consistently fail?

@luna215
Copy link
Author

luna215 commented Jan 5, 2024 •
edited

@srikanthccv yeah it happens consistently. I actually can install signoz 0.32.2 if I only have these settings on my yaml file:

global:
  storageClass: gp2-resizable
  cloud: aws

clickhouse:
  installCustomStorageClass: true

However, when I try to upgrade to add coldStorage with a separate yaml file that contains these configurations:

# cold-storage.yaml
clickhouse:
  installCustomStorageClass: true
  coldStorage:
    enabled: true
    defaultKeepFreeSpaceBytes: "10485760"
    endpoint: https://*******.s3.us-west-1.amazonaws.com/data/
    accessKey: ********
    secretAccess: **********
    role:
      enabled: true
      annotations:
        eks.amazonaws.com/role-arn: arn:aws:iam::***:role/***

using this command:
helm --namespace platform upgrade signoz signoz/signoz -f cold-storage.yaml
it gives me the same error as above and it breaks my Signoz - as I can't access it anymore and I just get a blank page with the text Something went wrong

@prashant-shahi
Copy link
Member

In most cases of migrator job failure, it means either the clickhouse or zookepeer was not healthy.

It would be best to watch the pods in another terminal with kubectl get pods -n platform and do share logs of the failure migrator pod.

However, when I try to upgrade to add coldStorage with a separate yaml file that contains these configurations:

Also, in case of new override values - you are required to merge the yaml with the previously applied one, was that done?

global:
  storageClass: gp2-resizable
  cloud: aws

clickhouse:
  installCustomStorageClass: true
  coldStorage:
    enabled: true
    defaultKeepFreeSpaceBytes: "10485760"
    endpoint: https://*******.s3.us-west-1.amazonaws.com/data/
    accessKey: ********
    secretAccess: **********
    role:
      enabled: true
      annotations:
        eks.amazonaws.com/role-arn: arn:aws:iam::***:role/***

And lastly, I see you have both access/secret key as well as role set up. You will only need one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@luna215 @prashant-shahi @srikanthccv