Support TLS Grpc communication between clusters. #11549

stone-98 · 2023-12-24T17:25:54Z

What is the purpose of the change

Currently, Nacos only supports TLS communication between SDK and Server, and there is a need to support communication between Server instances.

Brief changelog

Overview of Previous Logic

In the original system, only gRPC TLS communication from the SDK to the server was supported, with TLS configurations divided into client (RpcClientTlsConfig) and server (RpcServerTlsConfig) parts. The client had three dependencies on TLS configurations, while the server relied on a protocol negotiator to build TLS configurations.

Dependency Diagram:

Adjustments Made

TLS Configuration Logic Adjusted to Factory Pattern: Introduced the RpcTlsConfigFactory interface and its implementation classes to create TLS configurations according to the specific requirements of clients and servers.
Adjustment of Protocol Negotiator's Singleton Mode: Abstracted the singleton mode of the protocol negotiator into AbstractProtocolNegotiatorBuilderSingleton, and implemented SdkProtocolNegotiatorBuilderSingleton and ClusterProtocolNegotiatorBuilderSingleton based on it, for SDK and cluster protocol negotiation, respectively. Support for TLS gRPC interaction in clusters was added.
Adjustment of SSL Context Refresher: RpcServerSslContextRefresherHolder was adjusted to include support for cluster SSL context refresh, enhancing security.
Addition of Configuration Items: Introduced a series of configuration items specifically for cluster TLS communication.

Dependency Diagram:

Verifying this change

Demonstrating a three-node cluster

Configuring cluster.conf

The content of cluster.conf is as follows:

192.168.6.26:8848
192.168.6.26:8850
192.168.6.26:8852

Start Nacos on ports 8848, 8850, and 8852 respectively.

Generating certificates & signing

#!/bin/bash

# Generating a self-signed root certificate (CA)
openssl req -x509 -newkey rsa:4096 -days 365 -nodes \
    -keyout ca-key.pem -out ca-cert.pem \
    -subj "/C=CN/ST=Hunan/L=Changsha/O=nacos/OU=nacos/CN=nacos.io/[email protected]"

# Generating certificate requests and private keys for each member
for member in member1 member2 member3; do
    openssl req -newkey rsa:4096 -nodes \
        -keyout "${member}-key.pem" -out "${member}-req.pem" \
        -subj "/C=CN/ST=Hunan/L=Changsha/O=nacos/OU=nacos/CN=nacos.io/emailAddress=${member}@gmail.com"
done

# Signing each member's certificate request with the root certificate
for member in member1 member2 member3; do
    openssl x509 -req -in "${member}-req.pem" -days 60 \
        -CA ca-cert.pem -CAkey ca-key.pem -CAcreateserial \
        -out "${member}-cert.pem"
done

The generated files are as follows:

[root@yg-itom-test2 ca1]# ll
Total 76
-rw-r--r--. 1 root root 2090 Apr 12 17:24 ca-cert.pem		———— Self-signed root certificate file.
-rw-r--r--. 1 root root   17 Apr 12 17:24 ca-cert.srl		———— Root certificate serial number file, used to manage certificate serial numbers.
-rw-r--r--. 1 root root 3272 Apr 12 17:24 ca-key.pem		———— Private key file for the self-signed root certificate.
-rw-r--r--. 1 root root 1976 Apr 12 17:24 client-cert.pem	———— Certificate file for the SDK client.
-rw-r--r--. 1 root root 3272 Apr 12 17:24 client-key.pem	———— Private key file for the SDK client.
-rw-r--r--. 1 root root 1736 Apr 12 17:24 client-req.pem	———— Certificate request file for the SDK client.
-rwxrwxrwx. 1 root root 1541 Apr 12 15:16 createCa.sh		———— Bash script for creating certificates.
-rw-r--r--. 1 root root 1976 Apr 12 17:24 member1-cert.pem	———— Certificate file for member1.
-rw-r--r--. 1 root root 3272 Apr 12 17:24 member1-key.pem	———— Private key file for member1.
-rw-r--r--. 1 root root 1740 Apr 12 17:24 member1-req.pem	———— Certificate request file for member1.
-rw-r--r--. 1 root root 1976 Apr 12 17:24 member2-cert.pem	———— Certificate file for member2.
-rw-r--r--. 1 root root 3272 Apr 12 17:24 member2-key.pem	———— Private key file for member2.
-rw-r--r--. 1 root root 1740 Apr 12 17:24 member2-req.pem	———— Certificate request file for member2.
-rw-r--r--. 1 root root 1976 Apr 12 17:24 member3-cert.pem	———— Certificate file for member3.
-rw-r--r--. 1 root root 3272 Apr 12 17:24 member3-key.pem	———— Private key file for member3.
-rw-r--r--. 1 root root 1740 Apr 12 17:24 member3-req.pem	———— Certificate request file for member3.
-rw-r--r--. 1 root root 1976 Apr 12 17:24 server-cert.pem	———— Certificate file for the server.
-rw-r--r--. 1 root root 3272 Apr 12 17:24 server-key.pem	———— Private key file for the server.
-rw-r--r--. 1 root root 1736 Apr 12 17:24 server-req.pem	———— Certificate request file for the server.

Starting three Nacos instances

Start each Nacos instance with the following specified parameters:

bashCopy code# Nacos instance 1
-Dnacos.inetutils.ip-address=192.168.6.26
-Dnacos.core.member.lookup.type=file
-DembeddedStorage=true
-Dnacos.home=C:/Users/admin/nacos/cluster1
-Dserver.port=8848
-Dnacos.remote.peer.rpc.tls.enableTls=true
-Dnacos.remote.peer.rpc.tls.compatibility=false
-Dnacos.remote.peer.rpc.tls.certChainFile=ca/member1-cert.pem
-Dnacos.remote.peer.rpc.tls.certPrivateKey=ca/member1-key.pem
-Dnacos.remote.peer.rpc.tls.trustCollectionCertFile=ca/ca-cert.pem

# Nacos instance 2
-Dnacos.inetutils.ip-address=192.168.6.26
-Dnacos.core.member.lookup.type=file
-DembeddedStorage=true
-Dnacos.home=C:/Users/admin/nacos/cluster2
-Dserver.port=8850
-Dnacos.remote.peer.rpc.tls.enableTls=true
-Dnacos.remote.peer.rpc.tls.compatibility=false
-Dnacos.remote.peer.rpc.tls.certChainFile=ca/member2-cert.pem
-Dnacos.remote.peer.rpc.tls.certPrivateKey=ca/member2-key.pem
-Dnacos.remote.peer.rpc.tls.trustCollectionCertFile=ca/ca-cert.pem

# Nacos instance 3
-Dnacos.inetutils.ip-address=192.168.6.26
-Dnacos.core.member.lookup.type=file
-DembeddedStorage=true
-Dnacos.home=C:/Users/admin/nacos/cluster3
-Dserver.port=8852
-Dnacos.remote.peer.rpc.tls.enableTls=true
-Dnacos.remote.peer.rpc.tls.compatibility=false
-Dnacos.remote.peer.rpc.tls.certChainFile=ca/member3-cert.pem
-Dnacos.remote.peer.rpc.tls.certPrivateKey=ca/member3-key.pem
-Dnacos.remote.peer.rpc.tls.trustCollectionCertFile=ca/ca-cert.pem

Result

The cluster is running successfully:

Follow this checklist to help us incorporate your contribution quickly and easily:

Make sure there is a Github issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a Github issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
Format the pull request title like [ISSUE #123] Fix UnknownException when host config not exist. Each commit in the pull request should have a meaningful subject line and body.
Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
Write necessary unit-test to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in test module.
Run mvn -B clean package apache-rat:check findbugs:findbugs -Dmaven.test.skip=true to make sure basic checks pass. Run mvn clean install -DskipITs to make sure unit-test pass. Run mvn clean test-compile failsafe:integration-test to make sure integration-test pass.

stone-98 · 2023-12-24T17:36:47Z

@KomachiSion Please take a preliminary look to see if the implemented approach is correct. If it is, I will proceed to refine the code and enhance unit tests.

codecov-commenter · 2023-12-24T17:48:13Z

Codecov Report

Attention: Patch coverage is 71.42857% with 74 lines in your changes are missing coverage. Please review.

Project coverage is 67.83%. Comparing base (15fbf92) to head (072f350).

Additional details and impacted files

@@              Coverage Diff              @@
##             develop   #11549      +/-   ##
=============================================
- Coverage      67.85%   67.83%   -0.02%     
- Complexity      8915     8918       +3     
=============================================
  Files           1237     1241       +4     
  Lines          40466    40518      +52     
  Branches        4291     4285       -6     
=============================================
+ Hits           27459    27487      +28     
- Misses         11038    11062      +24     
  Partials        1969     1969

Files	Coverage Δ
...alibaba/nacos/client/config/impl/ClientWorker.java	`76.40% <100.00%> (-0.17%)`	⬇️
...ient/naming/remote/gprc/NamingGrpcClientProxy.java	`98.32% <100.00%> (ø)`
...a/nacos/common/remote/client/RpcClientFactory.java	`95.83% <ø> (ø)`
...nacos/common/remote/client/RpcClientTlsConfig.java	`100.00% <ø> (ø)`
...s/common/remote/client/grpc/GrpcClusterClient.java	`100.00% <ø> (ø)`
...nacos/common/remote/client/grpc/GrpcSdkClient.java	`100.00% <ø> (ø)`
...ba/nacos/common/remote/tls/RpcServerTlsConfig.java	`0.00% <ø> (ø)`
...cos/core/cluster/remote/ClusterRpcClientProxy.java	`73.61% <100.00%> (ø)`
...baba/nacos/core/remote/grpc/GrpcClusterServer.java	`71.42% <100.00%> (-2.26%)`	⬇️
.../alibaba/nacos/core/remote/grpc/GrpcSdkServer.java	`72.72% <100.00%> (+12.72%)`	⬆️
... and 15 more

... and 1 file with indirect coverage changes

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 15fbf92...072f350. Read the comment docs.

DSSLP · 2024-01-03T13:22:59Z

May I ask if it can also support TLS RPC communication between raft cluster?

stone-98 · 2024-01-03T14:09:08Z

May I ask if it can also support TLS RPC communication between raft cluster?

This PR only provides support for GRPC TLS communication between cluster nodes and does not yet support TLS communication between Raft clusters.

KomachiSion · 2024-01-05T09:54:50Z

May I ask if it can also support TLS RPC communication between raft cluster?

it should be submit to jraft community

KomachiSion · 2024-01-05T09:55:13Z

common/src/main/java/com/alibaba/nacos/common/remote/client/RpcClientFactory.java

@@ -34,11 +34,11 @@
 * @version $Id: RpcClientFactory.java, v 0.1 2020年07月14日 3:41 PM liuzunfei Exp $
 */
 public class RpcClientFactory {
- 
+


indent problem.
Please use nacos code style to reformat.

…lop-#issue-11456

stone-98 · 2024-02-02T10:15:04Z

@KomachiSion @shiyiyue1102 Please review.

…ract ProtocolNegotiatorBuilderSingleton.

# Conflicts: # common/src/test/java/com/alibaba/nacos/common/utils/ExceptionUtilTest.java

KomachiSion · 2024-03-12T08:06:37Z

common/src/main/java/com/alibaba/nacos/common/remote/tls/RpcServerTlsConfig.java

 *
- * @author githubcheng2978.
+ * @author stone-98


不要修改作者，可以添加一行author

改错了🤣，我改回来。

KomachiSion · 2024-03-12T08:09:17Z

common/src/main/java/com/alibaba/nacos/common/remote/tls/RpcServerTlsConfig.java

@@ -14,53 +14,60 @@
 * limitations under the License.
 */

-package com.alibaba.nacos.core.remote.tls;
+package com.alibaba.nacos.common.remote.tls;


Server的其实可以不动，继续放在core里。

放common里，但是client用不到这个类，是在增加client的大小

但是client中依赖RpcTlsFactory创建TlsClientConfig，所以RpcTlsFactory必须放在common，而RpcTlsFactory又直接依赖它，如果放在core将引用不到。

那应该把RpcTlsFactory进一步拆分，而且我没找到RpcTlsFactory这个类在哪。。

是RpcTlsConfigFactory，你的意思是要将他拆分成client的factory和cluster的factory是吗？然后分别位于common和core里面

KomachiSion · 2024-03-12T08:14:35Z

core/src/main/java/com/alibaba/nacos/core/remote/grpc/GrpcSdkServer.java

- /**
- * reload ssl context.
- */
- public void reloadProtocolNegotiator() {


为什么要删除这个？

我理解因为之前仅仅支持sdk server tls，所以将它放在GrpcSdkServer，但是新增cluster server tls后，我将它下沉到BaseRpcServer中去了。

我理解因为之前仅仅支持sdk server tls，所以将它放在GrpcSdkServer，但是新增cluster server tls后，我将它下沉到BaseRpcServer中去了。

没看到BaseRpcServer中有这个method

说错了是BaseGrpcServer

KomachiSion · 2024-04-08T02:45:54Z

common/src/main/java/com/alibaba/nacos/common/remote/tls/RpcServerTlsConfig.java

@@ -14,53 +14,60 @@
 * limitations under the License.
 */

-package com.alibaba.nacos.core.remote.tls;
+package com.alibaba.nacos.common.remote.tls;


那应该把RpcTlsFactory进一步拆分，而且我没找到RpcTlsFactory这个类在哪。。

KomachiSion · 2024-04-08T02:47:00Z

core/src/main/java/com/alibaba/nacos/core/remote/grpc/GrpcSdkServer.java

- /**
- * reload ssl context.
- */
- public void reloadProtocolNegotiator() {


我理解因为之前仅仅支持sdk server tls，所以将它放在GrpcSdkServer，但是新增cluster server tls后，我将它下沉到BaseRpcServer中去了。

没看到BaseRpcServer中有这个method

stone-98 · 2024-04-10T14:30:50Z

我把RpcTlsConfigFactory拆分为client和server的，然后调整cluster的参数指定为"nacos.remote.peer.rpc.tls"

…mote.peer.rpc.tls".

# Conflicts: # client/src/main/java/com/alibaba/nacos/client/config/impl/ClientWorker.java

…op-#issue-11456

stone-98 added 10 commits August 7, 2023 15:20

Fix exception code error.(alibaba#10925)

9ecd699

Merge branch 'develop' of https://github.com/alibaba/nacos into develop

dfd6139

[ISSUE alibaba#11456]Add RpcClusterClientTlsConfig.java.

c4a030d

[ISSUE alibaba#11456]Add cluster rpc tls config.

284c315

[ISSUE alibaba#11456]Add RpcClusterClientTlsConfig UT.

7ded5ba

[ISSUE alibaba#11456]Add cluster server tls.

57f9d40

[ISSUE alibaba#11456]Remove supportCommunicationTypes.

0eb0159

[ISSUE alibaba#11456]Fix unit testing and indentation handling

e1a755d

[ISSUE alibaba#11456]Indentation handling

2498821

[ISSUE alibaba#11456]Fix unit test and rpc constants.

153ad0a

[ISSUE alibaba#11456]Fix unit test.

3851794

stone-98 added 10 commits December 25, 2023 11:00

[ISSUE alibaba#11456]Optimize code.

51801ee

[ISSUE alibaba#11456]Fix check style.

b189613

[ISSUE alibaba#11456]Add unit test.

0e2532b

[ISSUE alibaba#11456]Fix check style.

76b48ff

[ISSUE alibaba#11456]Update unit test.

898cb40

[ISSUE alibaba#11456]Fix unit test.

be40ccd

[ISSUE alibaba#11456]Add License.

6ff0122

[ISSUE alibaba#11456]Fix unit test.

7329b6c

[ISSUE alibaba#11456]Fix unit test.

4a31a7a

[ISSUE alibaba#11456]Rename class.

24a43f6

KomachiSion requested changes Jan 5, 2024

View reviewed changes

KomachiSion requested a review from shiyiyue1102 January 5, 2024 09:56

stone-98 added 2 commits January 7, 2024 22:47

[ISSUE alibaba#11456]Optimize code.

911f81e

[ISSUE alibaba#11456]Handling indentation issues.

14134c1

Merge branch 'develop' of https://github.com/stone-98/nacos into deve…

3aafce3

…lop-#issue-11456

stone-98 and others added 3 commits February 8, 2024 23:23

Merge branch 'develop' into develop-#issue-11456

66dd055

[ISSUE alibaba#11456]Resolve conflict.

9f02f4e

[ISSUE alibaba#11456]Remove ProtocolNegotiatorBuilderManager and abst…

b5971a7

…ract ProtocolNegotiatorBuilderSingleton.

KomachiSion added this to the 2.4.0 milestone Feb 27, 2024

stone-98 added 3 commits March 12, 2024 08:41

[ISSUE alibaba#11456]Remove CommunicationType.java.

5a0c9d7

Merge branch 'develop' into develop-#issue-11456

6f71327

# Conflicts: # common/src/test/java/com/alibaba/nacos/common/utils/ExceptionUtilTest.java

[ISSUE alibaba#11456]Optimize code.

ae72fbc

KomachiSion requested changes Mar 12, 2024

View reviewed changes

[ISSUE alibaba#11456]Revert author.

072f350

stone-98 requested a review from KomachiSion March 12, 2024 10:23

stone-98 changed the title ~~Support TLS RPC communication between clusters.~~ Support TLS Grpc communication between clusters. Apr 3, 2024

KomachiSion requested changes Apr 8, 2024

View reviewed changes

stone-98 added 2 commits April 8, 2024 11:55

Splitting RpcTlsConfigFactory.

9b0bec3

Split RpcConstants.

12f1652

Divided RpcTlsConfigFactory, adjusted cluster parameters to "nacos.re…

0fa28f0

…mote.peer.rpc.tls".

stone-98 requested a review from KomachiSion April 18, 2024 02:56

stone-98 added 5 commits April 23, 2024 09:39

Merge branch 'develop' into develop-#issue-11456

13d4409

# Conflicts: # client/src/main/java/com/alibaba/nacos/client/config/impl/ClientWorker.java

check style.

e2ac17d

check style.

fa689c0

Merge branch 'develop' of https://github.com/alibaba/nacos into devel…

2165ee1

…op-#issue-11456

unit test.

82b0510

stone-98 mentioned this pull request May 11, 2024

nacos all port support tls #12088

Closed

KomachiSion added area/Test kind/feature type/feature and removed area/Test labels May 15, 2024

KomachiSion approved these changes May 15, 2024

View reviewed changes

KomachiSion merged commit 5169f06 into alibaba:develop May 15, 2024
7 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support TLS Grpc communication between clusters. #11549

Support TLS Grpc communication between clusters. #11549

stone-98 commented Dec 24, 2023 •

edited by KomachiSion

stone-98 commented Dec 24, 2023

codecov-commenter commented Dec 24, 2023 •

edited

DSSLP commented Jan 3, 2024

stone-98 commented Jan 3, 2024 •

edited

KomachiSion commented Jan 5, 2024

KomachiSion Jan 5, 2024

stone-98 Jan 10, 2024

stone-98 commented Feb 2, 2024

KomachiSion Mar 12, 2024

stone-98 Mar 12, 2024

KomachiSion Mar 12, 2024

KomachiSion Mar 12, 2024

stone-98 Mar 12, 2024

KomachiSion Apr 8, 2024

stone-98 Apr 8, 2024

stone-98 Apr 8, 2024

KomachiSion Mar 12, 2024

stone-98 Mar 12, 2024

KomachiSion Apr 8, 2024

stone-98 Apr 8, 2024

KomachiSion Apr 8, 2024

KomachiSion Apr 8, 2024

stone-98 commented Apr 10, 2024

Support TLS Grpc communication between clusters. #11549

Support TLS Grpc communication between clusters. #11549

Conversation

stone-98 commented Dec 24, 2023 • edited by KomachiSion

What is the purpose of the change

Brief changelog

Overview of Previous Logic

Adjustments Made

Verifying this change

Demonstrating a three-node cluster

Configuring cluster.conf

Generating certificates & signing

Starting three Nacos instances

Result

stone-98 commented Dec 24, 2023

codecov-commenter commented Dec 24, 2023 • edited

Codecov Report

DSSLP commented Jan 3, 2024

stone-98 commented Jan 3, 2024 • edited

KomachiSion commented Jan 5, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stone-98 commented Feb 2, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stone-98 commented Apr 10, 2024

stone-98 commented Dec 24, 2023 •

edited by KomachiSion

codecov-commenter commented Dec 24, 2023 •

edited

stone-98 commented Jan 3, 2024 •

edited