[Question] [Defrag and Compaction] - If Defrag and Compaction are "stop the world" operation

[jujiale]

hello, I am a novice in etcd， I want to know if the etcd's compact and defrag is like the java's gc which could cause the "stop the world" scenario, "stop the world" means when compact or defrag occurs, the etcd could not provided server(such as put del get ... ) for a while?

[vivekpatani]

According to my knowledge.

Compaction

Compaction is not a "stop the world" operation. But still should be used only after understanding the impact of compaction.

Defragmentation

Defragmentation is a "stop the world" operation.

As quoted from documentation

Note that defragmentation to a live member blocks the system from reading and writing data while rebuilding its states.
Note that defragmentation request does not get replicated over cluster. That is, the request is only applied to the local node. Specify all members in --endpoints flag or --cluster flag to automatically find all cluster members.

All these answers are only true for v3.5, for older versions you might find the answer in the source below
Source

[jujiale]

Note that defragmentation to a live member blocks the system from reading and writing data while rebuilding its states

@vivekpatani thanks for your reply, as the official artical say as above, I want to know if the "system" is refer to "etcd server".

[jmhbnz]

Hi @jujiale - Yes the system refere to the etcd server instance being defragmented.

[jujiale]

@jmhbnz thanks for your reply. I have one more question, as the official doc say， there are three metrics info indicate the space.
etcd_server_quota_backend_bytes , etcd_mvcc_db_total_size_in_bytes, etcd_mvcc_db_total_size_in_use_in_bytes
I want to know which scenario the alarm:NOSPACE could happen.
if both etcd_mvcc_db_total_size_in_bytes and etcd_mvcc_db_total_size_in_use_in_bytes size is approach the etcd_server_quota_backend_bytes, or just the etcd_mvcc_db_total_size_in_bytes approach the etcd_server_quota_backend_bytes, the alarm:NOSPACE may happen

[jujiale]

I have got the answer，thanks for all of you

[jujiale]

thank you for all for your reply. I finally find in etcd official doc, https://etcd.io/blog/2023/how_to_debug_large_db_size_issue/#compaction--defragmentation,there is some useful information in below:

1.Compaction is a cluster-wide operation, so you only need to execute compaction once on whichever etcd member. Of course, it will not do any harm if executing it multiple times.

2.Defragmentation is a time-consuming task, so it’s recommended to do it for each member one by one.

3.Defragmentation is an expensive operation, so you should do it as infrequent as possible. On the other hand, you also need to make sure any etcd member will not run out of the storage quota. etcdctl defrag [flags] can be used for defragmentation, but tools such as etcd-defrag can provide enhanced functionality and ease of use and should be considered to use.

4.There is a known issue that etcd might run into data inconsistency issue if it crashes in the middle of an online defragmentation operation using etcdctl or clientv3 API. All the existing v3.5 releases are affected, including 3.5.0 ~ 3.5.5. So please use etcdutl to offline perform defragmentation operation, but this requires taking each member offline one at a time. It means that you need to stop each etcd instance firstly, then perform defragmentation using etcdutl, start the instance at last. Please refer to the issue 1 in public <statement.