Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[bitnami/opensearch] master node not initializing #25689

Open
brianhong1117 opened this issue May 13, 2024 · 1 comment
Open

[bitnami/opensearch] master node not initializing #25689

brianhong1117 opened this issue May 13, 2024 · 1 comment
Assignees
@migruiz4
Labels
in-progress opensearch tech-issues The user has a technical issue about an application

Comments

@brianhong1117
Copy link

Name and Version

bitnami/opensearch:1.0.3

What architecture are you using?

amd64

What steps will reproduce the bug?

  1. Install chart with custom value as follows
  2. Wait for Node start
  3. Master-Node not initializing

Are you using any custom parameters or values?

security:
  ## @param security.enabled Enable X-Pack Security settings
  ##
  enabled: true
  ## @param security.adminPassword Password for 'admin' user
  ## Ref: https://github.com/bitnami/containers/tree/main/bitnami/opensearch#security
  ##
  adminPassword: '!@MyPassword'
  ## @param security.logstashPassword Password for Logstash
  ##
  logstashPassword: '!@MyPassword'
  ## @param security.existingSecret Name of the existing secret containing the OpenSearch password and
  ##
  existingSecret: ''
  ## FIPS mode
  ## @param security.fipsMode Configure opensearch with FIPS 140 compliant mode
  ## Ref: https://www.open.co/guide/en/opensearch/reference/current/fips-140-compliance.html
  ##
  fipsMode: false
  ## TLS configuration
  ##
  tls:
    ## @section OpenSearch admin parameters
    ## @param security.tls.admin.existingSecret Existing secret containing the certificates for admin
    ## @param security.tls.admin.certKey Key containing the crt for admin certificate (defaults to admin.crt)
    ## @param security.tls.admin.keyKey Key containing the key for admin certificate (defaults to admin.key)
    ##
    admin:
      existingSecret: ''
      certKey: ''
      keyKey: ''
    ## @param security.tls.restEncryption Enable SSL/TLS encryption for OpenSearch REST API.
    ##
    restEncryption: true
    ## @param security.tls.autoGenerated Create self-signed TLS certificates.
    ## NOTE: If autoGenerated certs are enabled and a new node type is enabled using helm upgrade, make sure you remove previously existing OpenSearch TLS secrets.
    ## Otherwise, the new node certs won't match the existing certs.
    ##
    autoGenerated: true
    ## @param security.tls.verificationMode Verification mode for SSL communications.
    ## Supported values: full, certificate, none.
    ## Ref: https://www.open.co/guide/en/opensearch/reference/current/security-settings.html
    ##
    verificationMode: 'full'
    ## TLS configuration for master nodes
    ##
    master:
      ## @param security.tls.master.existingSecret Existing secret containing the certificates for the master nodes
      ## @param security.tls.master.certKey Key containing the crt for master nodes certificate (defaults to tls.crt)
      ## @param security.tls.master.keyKey Key containing the key for master nodes certificate (defaults to tls.key)
      ## @param security.tls.master.caKey Key containing the ca for master nodes certificate (defaults to ca.crt)
      ##
      existingSecret: ''
      certKey: ''
      keyKey: ''
      caKey: ''
    ## TLS configuration for data nodes
    ##
    data:
      ## @param security.tls.data.existingSecret Existing secret containing the certificates for the data nodes
      ## @param security.tls.data.certKey Key containing the crt for data nodes certificate (defaults to tls.crt)
      ## @param security.tls.data.keyKey Key containing the key for data nodes certificate (defaults to tls.key)
      ## @param security.tls.data.caKey Key containing the ca for data nodes certificate (defaults to ca.crt)
      ##
      existingSecret: ''
      certKey: ''
      keyKey: ''
      caKey: ''
    ## TLS configuration for ingest nodes
    ##
    ingest:
      ## @param security.tls.ingest.existingSecret Existing secret containing the certificates for the ingest nodes
      ## @param security.tls.ingest.certKey Key containing the crt for ingest nodes certificate (defaults to tls.crt)
      ## @param security.tls.ingest.keyKey Key containing the key for ingest nodes certificate (defaults to tls.key)
      ## @param security.tls.ingest.caKey Key containing the ca for ingest nodes certificate (defaults to ca.crt)
      ##
      existingSecret: ''
      certKey: ''
      keyKey: ''
      caKey: ''
    ## TLS configuration for coordinating nodes
    ##
    coordinating:
      ## @param security.tls.coordinating.existingSecret Existing secret containing the certificates for the coordinating nodes
      ## @param security.tls.coordinating.certKey Key containing the crt for coordinating nodes certificate (defaults to tls.crt)
      ## @param security.tls.coordinating.keyKey Key containing the key for coordinating nodes certificate (defaults to tls.key)
      ## @param security.tls.coordinating.caKey Key containing the ca for coordinating nodes certificate (defaults to ca.crt)
      ##
      existingSecret: ''
      certKey: ''
      keyKey: ''
      caKey: ''
    ## @param security.tls.keystoreFilename Name of the keystore file
    ##
    keystoreFilename: opensearch.keystore.jks
    ## @param security.tls.truststoreFilename Name of the truststore
    ##
    truststoreFilename: opensearch.truststore.jks
    ## @param security.tls.usePemCerts Use this variable if your secrets contain PEM certificates instead of JKS/PKCS12
    ## Ignored when using autoGenerated certs.
    ##
    usePemCerts: false
    ## @param security.tls.passwordsSecret Existing secret containing the Keystore and Truststore passwords, or key password if PEM certs are used
    ##
    passwordsSecret: ''
    ## @param security.tls.keystorePassword Password to access the JKS/PKCS12 keystore or PEM key when they are password-protected.
    ## Ignored if security.tls.passwordsSecret is provided.
    ##
    keystorePassword: ''
    ## @param security.tls.truststorePassword Password to access the JKS/PKCS12 truststore when they are password-protected.
    ## Ignored if security.tls.passwordsSecret is provided.
    ##
    truststorePassword: ''
    ## @param security.tls.keyPassword Password to access the PEM key when they are password-protected.
    ## Ignored if security.tls.passwordsSecret is provided.
    ##
    keyPassword: ''
    ## @param security.tls.secretKeystoreKey Name of the secret key containing the Keystore password
    ##
    secretKeystoreKey: ''
    ## @param security.tls.secretTruststoreKey Name of the secret key containing the Truststore password
    ##
    secretTruststoreKey: ''
    ## @param security.tls.secretKey Name of the secret key containing the PEM key password
    ##
    secretKey: ''
    ## @param security.tls.nodesDN A comma separated list of DN for nodes
    ## e.g. nodesDN: "O=Example CA,C=SE,UID=c-5ca04c9328c8208704310f7c2ed16414"
    ##
    ##
    nodesDN: ''
    ## @param security.tls.adminDN A comma separated list of DN for admins
    ##
    adminDN: ''

What is the expected behavior?

Should load opensearch-security files and run without error

What do you see instead?

[2024-05-13T05:04:26,288][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-master-0] Directory /opt/bitnami/opensearch/config has insecure file permissions (should be 0700)
[2024-05-13T05:04:26,288][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-master-0] File /opt/bitnami/opensearch/config/admin.key has insecure file permissions (should be 0600)
[2024-05-13T05:04:26,288][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-master-0] File /opt/bitnami/opensearch/config/jvm.options has insecure file permissions (should be 0600)
[2024-05-13T05:04:26,288][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-master-0] Directory /opt/bitnami/opensearch/config/certs has insecure file permissions (should be 0700)
[2024-05-13T05:04:26,288][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-master-0] File /opt/bitnami/opensearch/config/certs/admin.crt has insecure file permissions (should be 0600)
[2024-05-13T05:04:26,289][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-master-0] File /opt/bitnami/opensearch/config/certs/ca.crt has insecure file permissions (should be 0600)
[2024-05-13T05:04:26,289][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-master-0] File /opt/bitnami/opensearch/config/certs/tls.key has insecure file permissions (should be 0600)
[2024-05-13T05:04:26,289][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-master-0] File /opt/bitnami/opensearch/config/certs/tls.crt has insecure file permissions (should be 0600)
[2024-05-13T05:04:26,289][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-master-0] File /opt/bitnami/opensearch/config/certs/admin.key has insecure file permissions (should be 0600)
[2024-05-13T05:04:26,290][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-master-0] File /opt/bitnami/opensearch/config/certs/..data has insecure file permissions (should be 0600)
[2024-05-13T05:04:26,290][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-master-0] Directory /opt/bitnami/opensearch/config/certs/..2024_05_13_05_00_29.2038276301 has insecure file permissions (should be 0700)
[2024-05-13T05:04:26,290][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-master-0] Directory /opt/bitnami/opensearch/config/opensearch-performance-analyzer has insecure file permissions (should be 0700)
[2024-05-13T05:04:26,290][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-master-0] File /opt/bitnami/opensearch/config/opensearch-performance-analyzer/plugin-stats-metadata has insecure file permissions (should be 0600)
[2024-05-13T05:04:26,290][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-master-0] File /opt/bitnami/opensearch/config/opensearch-performance-analyzer/opensearch_security.policy has insecure file permissions (should be 0600)
[2024-05-13T05:04:26,291][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-master-0] File /opt/bitnami/opensearch/config/opensearch-performance-analyzer/agent-stats-metadata has insecure file permissions (should be 0600)
[2024-05-13T05:04:26,291][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-master-0] File /opt/bitnami/opensearch/config/opensearch-performance-analyzer/rca_cluster_manager.conf has insecure file permissions (should be 0600)
[2024-05-13T05:04:26,291][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-master-0] File /opt/bitnami/opensearch/config/opensearch-performance-analyzer/log4j2.xml has insecure file permissions (should be 0600)
[2024-05-13T05:04:26,291][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-master-0] File /opt/bitnami/opensearch/config/opensearch-performance-analyzer/rca_idle_cluster_manager.conf has insecure file permissions (should be 0600)
[2024-05-13T05:04:26,291][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-master-0] File /opt/bitnami/opensearch/config/opensearch-performance-analyzer/supervisord.conf has insecure file permissions (should be 0600)
[2024-05-13T05:04:26,292][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-master-0] File /opt/bitnami/opensearch/config/opensearch-performance-analyzer/rca.conf has insecure file permissions (should be 0600)
[2024-05-13T05:04:26,292][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-master-0] File /opt/bitnami/opensearch/config/opensearch-performance-analyzer/performance-analyzer.properties has insecure file permissions (should be 0600)
[2024-05-13T05:04:26,292][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-master-0] Directory /opt/bitnami/opensearch/config/opensearch-reports-scheduler has insecure file permissions (should be 0700)
[2024-05-13T05:04:26,292][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-master-0] File /opt/bitnami/opensearch/config/opensearch-reports-scheduler/reports-scheduler.yml has insecure file permissions (should be 0600)
[2024-05-13T05:04:26,292][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-master-0] File /opt/bitnami/opensearch/config/opensearch_config.sample has insecure file permissions (should be 0600)
[2024-05-13T05:04:26,293][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-master-0] Directory /opt/bitnami/opensearch/config/opensearch-notifications has insecure file permissions (should be 0700)
[2024-05-13T05:04:26,293][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-master-0] File /opt/bitnami/opensearch/config/opensearch-notifications/notifications.yml has insecure file permissions (should be 0600)
[2024-05-13T05:04:26,293][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-master-0] File /opt/bitnami/opensearch/config/opensearch.yml has insecure file permissions (should be 0600)
[2024-05-13T05:04:26,293][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-master-0] Directory /opt/bitnami/opensearch/config/jvm.options.d has insecure file permissions (should be 0700)
[2024-05-13T05:04:26,293][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-master-0] File /opt/bitnami/opensearch/config/jvm.options.d/heap.options has insecure file permissions (should be 0600)
[2024-05-13T05:04:26,294][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-master-0] File /opt/bitnami/opensearch/config/log4j2.properties has insecure file permissions (should be 0600)
[2024-05-13T05:04:26,294][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-master-0] Directory /opt/bitnami/opensearch/config/opensearch-security has insecure file permissions (should be 0700)
[2024-05-13T05:04:26,294][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-master-0] File /opt/bitnami/opensearch/config/opensearch-security/audit.yml has insecure file permissions (should be 0600)
[2024-05-13T05:04:26,294][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-master-0] File /opt/bitnami/opensearch/config/opensearch-security/roles_mapping.yml has insecure file permissions (should be 0600)
[2024-05-13T05:04:26,294][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-master-0] File /opt/bitnami/opensearch/config/opensearch-security/action_groups.yml has insecure file permissions (should be 0600)
[2024-05-13T05:04:26,295][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-master-0] File /opt/bitnami/opensearch/config/opensearch-security/opensearch.yml.example has insecure file permissions (should be 0600)
[2024-05-13T05:04:26,295][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-master-0] File /opt/bitnami/opensearch/config/opensearch-security/whitelist.yml has insecure file permissions (should be 0600)
[2024-05-13T05:04:26,295][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-master-0] File /opt/bitnami/opensearch/config/opensearch-security/nodes_dn.yml has insecure file permissions (should be 0600)
[2024-05-13T05:04:26,295][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-master-0] File /opt/bitnami/opensearch/config/opensearch-security/internal_users.yml has insecure file permissions (should be 0600)
[2024-05-13T05:04:26,295][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-master-0] File /opt/bitnami/opensearch/config/opensearch-security/tenants.yml has insecure file permissions (should be 0600)
[2024-05-13T05:04:26,296][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-master-0] File /opt/bitnami/opensearch/config/opensearch-security/allowlist.yml has insecure file permissions (should be 0600)
[2024-05-13T05:04:26,296][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-master-0] File /opt/bitnami/opensearch/config/opensearch-security/roles.yml has insecure file permissions (should be 0600)
[2024-05-13T05:04:26,296][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-master-0] File /opt/bitnami/opensearch/config/opensearch-security/config.yml has insecure file permissions (should be 0600)
[2024-05-13T05:04:26,296][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-master-0] Directory /opt/bitnami/opensearch/config/opensearch-notifications-core has insecure file permissions (should be 0700)
[2024-05-13T05:04:26,296][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-master-0] File /opt/bitnami/opensearch/config/opensearch-notifications-core/notifications-core.yml has insecure file permissions (should be 0600)
[2024-05-13T05:04:26,297][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-master-0] Directory /opt/bitnami/opensearch/config/opensearch-observability has insecure file permissions (should be 0700)
[2024-05-13T05:04:26,297][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-master-0] File /opt/bitnami/opensearch/config/opensearch-observability/observability.yml has insecure file permissions (should be 0600)
[2024-05-13T05:04:26,297][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-master-0] File /opt/bitnami/opensearch/config/log4j2.file.properties has insecure file permissions (should be 0600)

[2024-05-13T05:09:54,579][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-master-0] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-05-13T05:09:54,579][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-master-0] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-05-13T05:09:54,579][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-master-0] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-05-13T05:09:54,579][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-master-0] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-05-13T05:09:54,579][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-master-0] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-05-13T05:09:54,579][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-master-0] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-05-13T05:09:54,579][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-master-0] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-05-13T05:09:54,579][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-master-0] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-05-13T05:09:54,579][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-master-0] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-05-13T05:09:54,579][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-master-0] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-05-13T05:10:00,029][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-master-0] Not yet initialized (you may need to run securityadmin)

When enabled security plugin, master node show following error.

Additional information

No response
@brianhong1117 brianhong1117 added the tech-issues The user has a technical issue about an application label May 13, 2024
@github-actions github-actions bot added the triage Triage is needed label May 13, 2024
@github-actions github-actions bot removed the triage Triage is needed label May 13, 2024
@github-actions github-actions bot assigned migruiz4 and unassigned javsalgar May 13, 2024
@migruiz4
Copy link
Member

Hi @brianhong1117,

Thank you for reporting this issue, I was able to reproduce and fix it.

The root cause of the issue was that we recently added default resources.requests and resource.limits. The Opensearch security settings requires the master-0 to bootstrap the security index, but at the time the process was started the jvm.options file had not yet been created, causing it to fail silently.

We have released a new version of the bitnami/opensearch fixing this (2.14.0-debian-12-r1) and the new version of the chart (1.0.9) should fix this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@migruiz4 migruiz4

Labels
in-progress opensearch tech-issues The user has a technical issue about an application
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@javsalgar @migruiz4 @brianhong1117