[Bug]: [ERROR] [funcutil/parallel.go:88] [loadRemoteFunc] [error="failed to read stats_log/xxxxx: NoSuchKey

[goldbridge18]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version:2.3.10
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka):    kafka
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

清理掉NoSuchKey 相关的 partition之后，load可以成功，再次写入时仍然报错：

[2024/05/06 01:06:19.071 +00:00] [WARN] [segments/segment_loader.go:512] ["load remote segment bloom filter failed"] [traceID=9e7da3308498a79d3e31e36954b64696] [collectionID=444366786892873263] [segmentIDs="[449410906289424449]"] [partitionID=449410906285749753] [segmentID=449410906289424449] [error="failed to read stats_log/444366786892873263/449410906285749753/449410906289424449/100/449410906289424603: NoSuchKey(key=stats_log/444366786892873263/449410906285749753/449410906289424449/100/449410906289424603)"]
[2024/05/06 01:06:19.071 +00:00] [ERROR] [funcutil/parallel.go:88] [loadRemoteFunc] [error="failed to read stats_log/444366786892873263/449410906285749753/449410906289424449/100/449410906289424603: NoSuchKey(key=stats_log/444366786892873263/449410906285749753/449410906289424449/100/449410906289424603)"] [idx=0] [stack="github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n\t/go/src/github.com/milvus-io/milvus/pkg/util/funcutil/parallel.go:88"]
[2024/05/06 01:06:19.071 +00:00] [WARN] [segments/segment_loader.go:527] ["failed to load remote segment"] [traceID=9e7da3308498a79d3e31e36954b64696] [collectionID=444366786892873263] [segmentIDs="[449410906289424449]"] [error="failed to read stats_log/444366786892873263/449410906285749753/449410906289424449/100/449410906289424603: NoSuchKey(key=stats_log/444366786892873263/449410906285749753/449410906289424449/100/449410906289424603)"]
[2024/05/06 01:06:19.071 +00:00] [WARN] [delegator/delegator_data.go:363] ["failed to load bloom filter set for segment"] [traceID=9e7da3308498a79d3e31e36954b64696] [collectionID=444366786892873263] [channel=by-dev-rootcoord-dml_9_444366786892873263v3] [replicaID=449565231102558210] [workID=878] [segments="[449410906289424449]"] [error="failed to read stats_log/444366786892873263/449410906285749753/449410906289424449/100/449410906289424603: NoSuchKey(key=stats_log/444366786892873263/449410906285749753/449410906289424449/100/449410906289424603)"]

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

milvus-logs20240506_01.tar.gz

Anything else?

No response

[xiaofan-luan]

cleanup this file stats_log/444366786892873263/449410906285749753/449410906289424449/100/449410906289424603 should work.
once it recovered, do a compaction on this collection to rebuild stats log

[xiaofan-luan]

cleanup this file stats_log/444366786892873263/449410906285749753/449410906289424449/100/449410906289424603 should work.
once it recovered, do a compaction on this collection to rebuild stats log

[xiaofan-luan]

cleanup this file stats_log/444366786892873263/449410906285749753/449410906289424449/100/449410906289424603 should work.
once it recovered, do a compaction on this collection to rebuild stats log

[yanliang567]

/assign @goldbridge18
please try as comments above and keep us posted

/unassign