Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: There are remaining segments that have not been merged compaction and GC #32765

Open
1 task done
ThreadDao opened this issue May 6, 2024 · 5 comments
Open
1 task done

[Bug]: There are remaining segments that have not been merged compaction and GC #32765

ThreadDao opened this issue May 6, 2024 · 5 comments
Assignees
@ThreadDao
Labels
kind/bug Issues or changes related a bug triage/accepted Indicates an issue or PR is ready to be actively worked on.
Milestone
2.4.2

Comments

@ThreadDao
Copy link
Contributor

ThreadDao commented May 6, 2024 •
edited

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version: master-20240429-ac82cef0
- Deployment mode(standalone or cluster): standalone
- MQ type(rocksmq, pulsar or kafka):  kafka  
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

  1. deploy standalone and enable L0 segment config
  2. create collection with 2 shards and a partition -> index -> insert 20m-128d data -> flush -> index again -> load
  3. concurrent: search + upsert + flush
  4. upgrade config: disable L0 segment and wait healthy
  5. concurrent: search + upsert + flush
  6. After a few days, the birdwatcher shows that there are still 371 small segments and 605 dropped segments
--- Growing: 0, Sealed: 0, Flushed: 394, Dropped: 605
--- Small Segments: 371, row count: 956795	 Other Segments: 23, row count: 19920412
--- Total Segments: 394, row count: 20877207

image

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

argo workflow: https://argo-workflows.zilliz.cc/archived-workflows/qa/62a9b6ae-64a6-4e73-83b3-397a2a8d2d8f?nodeId=zong-concurrent-upgrade-config-cron-1714402800-1423649165

pods:

enable-to-disable-op-47-8424-etcd-0                               1/1     Running       0               6d11h   10.104.26.125   4am-node32   <none>           <none>
enable-to-disable-op-47-8424-kafka-0                              1/1     Running       0               6d11h   10.104.26.131   4am-node32   <none>           <none>
enable-to-disable-op-47-8424-kafka-1                              1/1     Running       0               6d11h   10.104.25.72    4am-node30   <none>           <none>
enable-to-disable-op-47-8424-kafka-2                              1/1     Running       0               6d11h   10.104.32.238   4am-node39   <none>           <none>
enable-to-disable-op-47-8424-kafka-zookeeper-0                    1/1     Running       0               6d11h   10.104.26.132   4am-node32   <none>           <none>
enable-to-disable-op-47-8424-kafka-zookeeper-1                    1/1     Running       0               6d11h   10.104.34.83    4am-node37   <none>           <none>
enable-to-disable-op-47-8424-kafka-zookeeper-2                    1/1     Running       0               6d11h   10.104.25.74    4am-node30   <none>           <none>
enable-to-disable-op-47-8424-milvus-standalone-5b59758755-srgs5   1/1     Running       0               6d8h    10.104.27.141   4am-node31   <none>           <none>
enable-to-disable-op-47-8424-minio-77b664644f-kmp4n               1/1     Running       0               6d11h   10.104.26.128   4am-node32   <none>           <none>
enable-to-disable-op-47-8424-minio-update-prometheus-secreblkt7   0/1     Completed     0               6d11h   10.104.33.22    4am-node36   <none>           <none>

Anything else?

No response
@ThreadDao ThreadDao added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels May 6, 2024
@ThreadDao ThreadDao added this to the 2.4.1 milestone May 6, 2024
@XuanYang-cn
Copy link
Contributor

/assign

@yanliang567
Copy link
Contributor

@XuanYang-cn could be a bug injected by #32326, which was reverted in v2.4.1

@yanliang567 yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels May 6, 2024
@yanliang567 yanliang567 removed their assignment May 6, 2024
@XuanYang-cn
Copy link
Contributor

The choosing of the same segments making datanode Lock Segment twice, and block forever.

[...] ["...compaction"] [... ["segment IDs"="[449420485790442035,449420485790443250,449420485790443250]"]

XuanYang-cn added a commit to XuanYang-cn/milvus that referenced this issue May 7, 2024
@XuanYang-cn
fix: Compaction trigger choose 2 same segments
52cf9d3 
DataNode would stuck at compactor try to lock the
same segmentID

See also: milvus-io#32765

Signed-off-by: yangxuan <[email protected]>
@XuanYang-cn XuanYang-cn mentioned this issue May 7, 2024
Merged
@yanliang567 yanliang567 modified the milestones: 2.4.1, 2.4.2 May 7, 2024
sre-ci-robot pushed a commit that referenced this issue May 7, 2024
@XuanYang-cn
fix: Compaction trigger choose 2 same segments (#32800)
6843d6d 
DataNode would stuck at compactor try to lock the
same segmentID

See also: #32765

---------

Signed-off-by: yangxuan <[email protected]>
XuanYang-cn pushed a commit to XuanYang-cn/milvus that referenced this issue May 8, 2024
@tedxu @XuanYang-cn
enhance: [2.3]trigger compaction by size and fix (milvus-io#32326)
ba2ea28 
fix: Compaction trigger choose 2 same segments (milvus-io#32800)

DataNode would stuck at compactor try to lock the
same segmentID

See milvus-io#32328, milvus-io#32765

Co-Authored-by: Ted Xu <[email protected]>
Signed-off-by: yangxuan <[email protected]>
XuanYang-cn pushed a commit to XuanYang-cn/milvus that referenced this issue May 8, 2024
@tedxu @XuanYang-cn
enhance: [2.4]trigger compaction by size and fix (milvus-io#32326)
af6811f 
fix: Compaction trigger choose 2 same segments (milvus-io#32800)

DataNode would stuck at compactor try to lock the
same segmentID

See milvus-io#32328, milvus-io#32765

Co-Authored-by: Ted Xu <[email protected]>
Signed-off-by: yangxuan <[email protected]>
@XuanYang-cn XuanYang-cn mentioned this issue May 8, 2024
Merged
@XuanYang-cn
Copy link
Contributor

@ThreadDao Please help verify master branch, THX

sre-ci-robot pushed a commit that referenced this issue May 9, 2024
@XuanYang-cn @tedxu
enhance: [2.4]trigger compaction by size and fix (#32326) (#32846)
c0d9efa 
fix: Compaction trigger choose 2 same segments (#32800)

DataNode would stuck at compactor try to lock the
same segmentID

See: #32328, #32765
pr: #32800 #32326

Signed-off-by: yangxuan <[email protected]>
Co-authored-by: Ted Xu <[email protected]>
@XuanYang-cn
Copy link
Contributor

/assign @ThreadDao
/unassign
should be fixed, please help verify

@sre-ci-robot sre-ci-robot assigned ThreadDao and unassigned XuanYang-cn May 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ThreadDao ThreadDao

Labels
kind/bug Issues or changes related a bug triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
2.4.2
Development

No branches or pull requests
3 participants
@ThreadDao @XuanYang-cn @yanliang567