Trying to set up ingress with nginx

[sergio-moreira-everi]

Type of question

How to perform a particular operation

What did you do?

1. I have the helm operator installed;
   helm install k8ssandra-operator k8ssandra/k8ssandra-operator -n k8ssandra-operator --set global.clusterScoped=true
   For now I just want the operator and a k8ssandra cluster deployed in the same cluster.
2. I can successfully port forward to 9042 and use cqlsh

cqlsh 127.0.0.1 50672 -u <USERNAME> -p <PASSWORD>

3. So I tried setting up Ingress with nginx to point to the cluster service for port 9042

metadata:
  name: <CASS_INGRESS>
  namespace: <CASS_NS>
spec:
  ingressClassName: nginx
  rules:
  - host: <HOSTNAME>
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: <NAME>-dc1-service
            port:
              number: 9042

4. Now I try hitting the endpoint with

cqlsh <HOSTNAME> 80 -u <USERNAME> -p <PASSWORD>

and I get the following error:

Connection error: ('Unable to connect to any servers', {'<IP>:80': ProtocolError('This version of the driver does not support protocol version 72')})

Did you expect to see some different?
I'd expect to be able to run cqlsh just fine and enter the shell

Environment

• Helm charts version
  1.11.1

[bradfordcp]

HTTP ingress is not going to work here as Cassandra uses TCP. There are ways to do TCP ingress with nginx (and other ingress controllers). Typically it requires either adding a configmap to the ingress or using a custom resource definition provided via the ingress instead of the standard k8s ingress objects. (Note the Gateway API should help with this in the future).

Now when you get that configured you must decide how you route from the ingress to pods. The naive solution is to let it round robin, but this will introduce latency as the drivers prefer to connect to a node that houses the data for the query and use it as a coordinator. Additionally, should you run multiple C* clusters within a single k8s cluster how do you route to the different clusters? One option is to put each cluster on its own port within the ingress. Alternatively you could leverage TLS with SNI to allow for direct routing from the ingress to the appropriate node. It is worth noting that if you go this route there is additional configuration needed within the client application to configure TLS appropriately AND logic to update the routing within your ingress should you C* cluster topology change.

That's not to say it isn't possible, and we have some examples of how to accomplish this.

[bradfordcp]

So with all that being said what type of example would you prefer?

• TLS w/ SNI routing
• Port-based routing