Can stargate service as the contact points of cassandra?

[ZhiXingHeYiApple]

Bug Report

Describe the bug
I install the cassandra cluster by k8ssandra operator. The configuration describe as below:

cassandra:
  # Version of Apache Cassandra to deploy
  version: "4.0.1" #"3.11.10"

  # -- Security context override for cassandra container
  securityContext:
    # -- Mark root filesystem as read only
    #readOnlyRootFilesystem: false

    # -- Run cass-operator container as non-root user
    # runAsNonRoot: true

#    runAsNonRoot: true
#    # -- Group for the user running the cass-operator container / process
#    # runAsGroup: 65534
#
#    runAsGroup: 65534
#    # -- User for running the cass-operator container / process
#    # runAsUser: 65534
#    runAsUser: 65534

  # -- Security context override for pod where Cassandra container resides
  podSecurityContext:
    #readOnlyRootFilesystem: false
#    runAsNonRoot: true
#    runAsUser: 999
#    runAsGroup: 999
#    fsGroup: 999
    fsGroup: 999

  # Configuration for the /var/lib/cassandra mount point
  cassandraLibDirVolume:
    # AWS provides this storage class on EKS clusters out of the box. Note we
    # are using `gps` here as it has `volumeBindingMode: WaitForFirstConsumer`
    # which is important during scheduling.
    storageClass: gp3

    # The recommended live data size is 1 - 1.5 TB. A 2 TB volume supports this
    # much data along with room for compactions. Consider increasing this value
    # as the number of provisioned IOPs is directly related to the volume size.
    size: 256Gi  # 2048Gi
  allowMultipleNodesPerWorker: true
  heap:
    size: 8G  #31G
    newGenSize: 1G  #31G

  resources:
    requests:
      cpu: 4000m #7000m
      memory: 16Gi  #60Gi
    limits:
      cpu: 4000m #7000m
      memory: 16Gi  #60Gi

  # This key defines the logical topology of your cluster. The rack names and
  # labels should be updated to reflect the Availability Zones where your EKS
  # cluster is deployed.
  datacenters:
    - name: dc1
      size: 2 #3
      racks:
#        - name: us-east-1a
#          affinityLabels:
#            topology.kubernetes.io/zone: us-east-1a
        - name: cn-northwest-1b
          affinityLabels:
            topology.kubernetes.io/zone: cn-northwest-1b
        - name: cn-northwest-1c
          affinityLabels:
            topology.kubernetes.io/zone: cn-northwest-1c

kube-prometheus-stack:
  enabled: false
  global:
    imagePullSecrets:
      - name: harbor-stag-secret
  prometheusOperator:
    # Installs the Prometheus Operator, omitting this parameter will result in
    # resources not being deployed.
    enabled: true
    # -- Locks Prometheus operator to this namespace. Changing this setting may
    # result in a non-namespace scoped deployment.
    namespaces:
      releaseNamespace: true
      additional: [ ]
    # -- Monitoring of prometheus operator
    serviceMonitor:
      selfMonitor: false

    admissionWebhooks:
      patch:
        image:
          repository: registry-stag-hwy.bestsign.tech/search/ingress-nginx-kube-webhook-certgen  #k8s.gcr.io/ingress-nginx/kube-webhook-certgen   # 目前拉取不到 k8s.gcr.io的镜像
          tag: v1  #v1.0
          sha: ""
  prometheus:
    prometheusSpec:
      storageSpec:
        ## Using PersistentVolumeClaim
        ##
        volumeClaimTemplate:
          spec:
            storageClassName: gp3
            accessModes: ["ReadWriteOnce"]
            resources:
              requests:
                storage: 70Gi
        #    selector: {}
  grafana:
    adminUser: admin
    adminPassword: admin123
stargate:
  enabled: true
  replicas: 1
  heapMB: 512
  cpuReqMillicores: 200
  cpuLimMillicores: 1000

# Backup / Restore
#medusa:
#  enabled: true
#  storage: s3
#
#  # Reference the Terraform output for the correct bucket name to use here.
#  bucketName: prod-k8ssandra-s3-bucket
#
#  # The secret here must align with the value used in the previous section.
#  storageSecret: prod-k8ssandra-medusa-key
#
#  storage_properties:
#    region: us-east-1

When I want to connect to the cassandra cluster, I am confused that which endpoints should I select, I think stargate service address may be ok. But when I try it, I find that cql session can't acquire rowsmetadata about columns. So I get the exception about cluster_name is not a column in this row. And When I use cassandra node pod service address or pod ip as connect endpoint, it's OK.

To Reproduce
Steps to reproduce the behavior:
(1) client side code

<dependencies>
        <dependency>
            <groupId>com.datastax.oss</groupId>
            <artifactId>java-driver-core</artifactId>
            <version>4.6.1</version>
        </dependency>
    </dependencies>

public class CassandraConnector {

    private CqlSession session;

    public void connect(String node, Integer port, String dataCenter, String username, String pass) {
        CqlSessionBuilder builder = CqlSession.builder();
        builder.addContactPoint(new InetSocketAddress(node, port));
        builder.withLocalDatacenter(dataCenter);
        if(StringUtils.isNoneBlank(username) && StringUtils.isNoneBlank(pass)) builder.withAuthCredentials(username, pass);

        session = builder.build();
    }

    public CqlSession getSession() {
        return this.session;
    }

    public void close() {
        session.close();
    }
}

public class Test {
    public static void main(String[] args) {
        CassandraConnector cassandraConnector = new CassandraConnector();
       /** NORMAL:  10.203.61.230 is the cassandra pod ip**/
        cassandraConnector.connect("10.203.59.39", 9042, "dc1", "k8ssandra-superuser", "zKdevcLJhuCGjHRsc7CK");
       /**FAIL (row.getString("cluster_name") fail due to cluster_name is not a column in this row):       kubectl port-forward svc/k8ssandra-dc1-stargate-service 9042 -n k8ssandra **/
        cassandraConnector.connect("localhost", 9042, "dc1", "k8ssandra-superuser", "zKdevcLJhuCGjHRsc7CK");
        cassandraConnector.getSession().prepareAsync("select cluster_name, data_center, release_version from system.local")
                        .thenApply(ps -> ps.bind()).thenApply(bs -> {
                    CompletionStage<String> a = cassandraConnector.getSession().executeAsync(bs).thenApply(ars -> {
                        Row row = ars.one();
                        String clusterName = row.getString("cluster_name");
                        return clusterName;
                    });
                    return a;
                });

        try {
            Thread.sleep(1000000);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
        cassandraConnector.close();
    }
}

Expected behavior
Can stargate service address as the contact points of cassandra?

Screenshots
If applicable, add screenshots to help explain your problem.
Normal screenshots

Fail screenshot

[adejanovski]

@olim7t, could you take a look at this ticket and tell us what you think? Thanks!