A connection error occurs after TLS authentication is enabled

[Howie516]

When testing TLS, copy CA-related files in the server configs to the local PC, and apply after modifying the k8s configmap. After calling the SDK, an error was reported, and the network ping was successful

Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 000006040000000000000500004000
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1213) ~[netty-handler-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1283) ~[netty-handler-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) ~[netty-codec-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) ~[netty-codec-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) ~[netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658) ~[netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584) ~[netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[netty-transport-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[netty-common-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.72.Final.jar:4.1.72.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.72.Final.jar:4.1.72.Final]
	at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_382]

[Howie516]

After restarting pod, configmap appears to work, but you can't log in with your user name and password, and an error occurs when you log in with sdk tls:

Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:146) ~[?:1.8.0_382] at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:127) ~[?:1.8.0_382] at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) ~[?:1.8.0_382] at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:451) ~[?:1.8.0_382] at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:323) ~[?:1.8.0_382] at sun.security.validator.Validator.validate(Validator.java:271) ~[?:1.8.0_382] at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:315) ~[?:1.8.0_382] at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:278) ~[?:1.8.0_382] at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:141) ~[?:1.8.0_382] at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1334) ~[?:1.8.0_382] at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1231) ~[?:1.8.0_382] at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1174) ~[?:1.8.0_382] at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:377) ~[?:1.8.0_382] at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) ~[?:1.8.0_382] at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:981) ~[?:1.8.0_382] at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:968) ~[?:1.8.0_382] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_382] at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:915) ~[?:1.8.0_382] at io.netty.handler.ssl.SslHandler$SslTasksRunner.run(SslHandler.java:1785) ~[netty-handler-4.1.72.Final.jar:4.1.72.Final] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_382] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_382] at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_382]

[yhmo]

Follow the description of TLS example:

milvus-sdk-java/examples/main/java/io/milvus/TLSExample.java

Line 33 in 6bc4304

// Note: read the following description before running this example

[yhmo]

The following steps are an example of using docker-compose to launch a milvus cluster locally with tls configurations.

1. Download this zip file to a local folder, and extract it. There are docker-compose.yaml, milvus.yaml and a "tls" folder in it.
   233.zip

2. cd into the "tls" folder, generate the certification files

chmod +x gen.sh
./gen.sh

3. cd to the folder extracted by the 233.zip
   docker-compose up -d
   you will see a local cluster is started

4. use java sdk to connect the server

        ConnectParam connectParam = ConnectParam.newBuilder()
                .withHost("localhost")
                .withPort(19530)
                .withServerName("localhost")
                .withServerPemPath("[the extracted folder path]/tls/server.pem")
                .build();
        MilvusServiceClient milvusClient = new MilvusServiceClient(connectParam);

        R<CheckHealthResponse> health = milvusClient.checkHealth();
        if (health.getStatus() != R.Status.Success.getCode()) {
            throw new RuntimeException(health.getMessage());
        } else {
            System.out.println(health);
        }

[yhmo]

The key points:

• In the docker-compose.yaml, the milvus.yaml is mapped to the ourside milvus.yaml, the internal certification file path "/milvus/configs/cert" is mapped to the outside "tls" folder

  proxy:
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/milvus.yaml:/milvus/configs/milvus.yaml
      - ${DOCKER_VOLUME_DIRECTORY:-.}/tls:/milvus/configs/cert

• In the outside milvus.yaml, tls paths and tls mode is configurated:

common:
  security:
    tlsMode: 1 # 1 is one-way tls
tls:
  serverPemPath: /milvus/configs/cert/server.pem
  serverKeyPath: /milvus/configs/cert/server.key
  caPemPath: /milvus/configs/cert/ca.pem

• In the client java code, the correct file is specified:

ConnectParam connectParam = ConnectParam.newBuilder()
                .withHost("localhost")
                .withPort(19530)
                .withServerName("localhost")
                .withServerPemPath("[the extracted folder path]/tls/server.pem")

[yhmo]

If you need two-way tls:

1. set the tlsMode to 2 in the milvus.yaml:

common:
  security:
    tlsMode: 2

2. specify the required certification files in the client java code

ConnectParam connectParam = ConnectParam.newBuilder()
                .withHost("localhost")
                .withPort(19530)
                .withServerName("localhost")
                .withCaPemPath("[the extracted folder path]/tls/ca.pem")
                .withClientKeyPath("[the extracted folder path]/tls/client.key")
                .withClientPemPath("[the extracted folder path]/tls/client.pem")
                .build();
        MilvusServiceClient milvusClient = new MilvusServiceClient(connectParam);