Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Milvus standalone random crash with 100% CPU utilisation #32382

Open
1 task done
mihailyanchev opened this issue Apr 17, 2024 · 7 comments
Open
1 task done

[Bug]: Milvus standalone random crash with 100% CPU utilisation #32382

mihailyanchev opened this issue Apr 17, 2024 · 7 comments
Assignees
@mihailyanchev @yanliang567
Labels
kind/bug Issues or changes related a bug stale indicates no udpates for 30 days triage/needs-information Indicates an issue needs more information in order to work on it.

Comments

@mihailyanchev
Copy link

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version: 2.3.4
- Deployment mode(standalone or cluster): standalone
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2): pymilvus 2.3.1
- OS(Ubuntu or CentOS): Ubuntu 22.04.2 LTS
- CPU/Memory: AWS t3.xlarge instance (4 cores, model: Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz and 16GB RAM, 100GB gp3 storage) 
- GPU: no GPU
- Others:

Current Behavior

We experienced two random shutdowns of the milvus-standalone container during periods of larger load to the Milvus (mainly insert and delete operations). There are no errors in the logs immediately before the shutdown. We observe 100% CPU utilisation right before the crash and the first time it happened the whole machine crashed too and restarted automatically. The minio and the etcd containers remain alive and running.

We have around 2-4 collections with no more than 200-300K records in each.

The first time this happened it was impossible to return the Milvus-standalone to normal. Every time we restart the container it would go back to 100% cpu utilisation and crash again. After several restarts it managed to remain stable, but it never managed to load its collections, so we had to rebuild everything from scratch.

I am wondering whether the resources are too limited for our setup and whether we have to either allocate more compute resources or move to the distributed mode.

Expected Behavior

I would expect some error log as to what is happening or some graceful way to let us know it needs more resources.

Steps To Reproduce

I have not been able to reproduce intentionally. It seems it is related to a longer period (several hours) of inserts and consecutive deletions of previously inserted records. 

Our pipeline requires to insert some records for immediate use, but then later on delete these records and replace them with new ones, once a second stage of processing has been completed.

Milvus Log

The log for this was 1.7GB so I am just posting the very begging and the very end before the crash.

The beginning:

2024/04/02 09:07:03 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined

    __  _________ _   ____  ______    
   /  |/  /  _/ /| | / / / / / __/    
  / /|_/ // // /_| |/ / /_/ /\ \    
 /_/  /_/___/____/___/\____/___/     

Welcome to use Milvus!
Version:   v2.3.4
Built:     Tue Jan  2 07:34:57 UTC 2024
GitCommit: 7a192da8
GoVersion: go version go1.20.7 linux/amd64

open pid file: /run/milvus/standalone.pid
lock pid file: /run/milvus/standalone.pid
[2024/04/02 09:07:03.415 +00:00] [INFO] [roles/roles.go:306] ["starting running Milvus components"]
[2024/04/02 09:07:03.415 +00:00] [INFO] [roles/roles.go:169] ["Enable Jemalloc"] ["Jemalloc Path"=/milvus/lib/libjemalloc.so]
[2024/04/02 09:07:03.434 +00:00] [DEBUG] [config/etcd_source.go:49] ["init etcd source"] [etcdInfo="{\"UseEmbed\":false,\"UseSSL\":false,\"Endpoints\":[\"etcd:2379\"],\"KeyPrefix\":\"by-dev\",\"CertFile\":\"/path/to/etcd-client.pem\",\"KeyFile\":\"/path/to/etcd-client-key.pem\",\"CaCertFile\":\"/path/to/ca.pem\",\"MinVersion\":\"1.3\",\"RefreshInterval\":5000000000}"]
[2024/04/02 09:07:03.435 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[etcd:2379]"] [minVersion=1.3]
[2024/04/02 09:07:03.435 +00:00] [INFO] [config/refresher.go:66] ["start refreshing configurations"] [source=FileSource]
[2024/04/02 09:07:03.437 +00:00] [DEBUG] [config/etcd_source.go:141] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[etcd:2379]"]
[2024/04/02 09:07:05.326 +00:00] [INFO] [config/refresher.go:66] ["start refreshing configurations"] [source=EtcdSource]
[2024/04/02 09:07:05.329 +00:00] [INFO] [config/refresher.go:66] ["start refreshing configurations"] [source=FileSource]
[2024/04/02 09:07:05.329 +00:00] [INFO] [paramtable/hook_config.go:21] ["hook config"] [hook={}]
[2024/04/02 09:07:05.329 +00:00] [INFO] [roles/roles.go:255] [setupPrometheusHTTPServer]
[2024/04/02 09:07:05.329 +00:00] [INFO] [http/server.go:112] ["management listen"] [addr=:9091]
[2024/04/02 09:07:05.330 +00:00] [INFO] [rootcoord/root_coord.go:157] ["update rootcoord state"] [state=Abnormal]
[2024/04/02 09:07:05.330 +00:00] [DEBUG] [rootcoord/service.go:166] ["init params done.."]
[2024/04/02 09:07:05.330 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[etcd:2379]"] [minVersion=1.3]
[2024/04/02 09:07:05.330 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[etcd:2379]"] [minVersion=1.3]
[2024/04/02 09:07:05.330 +00:00] [INFO] [components/index_coord.go:38] ["IndexCoord running ..."]
[2024/04/02 09:07:05.331 +00:00] [DEBUG] [querynode/service.go:100] [QueryNode] [port=21123]
[2024/04/02 09:07:05.331 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[etcd:2379]"] [minVersion=1.3]
[2024/04/02 09:07:05.331 +00:00] [DEBUG] [rootcoord/service.go:183] ["etcd connect done ..."]
[2024/04/02 09:07:05.331 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[etcd:2379]"] [minVersion=1.3]
[2024/04/02 09:07:05.331 +00:00] [DEBUG] [rootcoord/service.go:243] ["start grpc "] [port=53100]
[2024/04/02 09:07:05.332 +00:00] [DEBUG] [querynode/service.go:117] ["QueryNode connect to etcd successfully"]
[2024/04/02 09:07:05.332 +00:00] [DEBUG] [datacoord/service.go:147] ["network port"] [port=13333]
[2024/04/02 09:07:05.331 +00:00] [INFO] [datanode/service.go:245] ["DataNode address"] [address=172.21.0.5:21124]
[2024/04/02 09:07:05.332 +00:00] [DEBUG] [indexnode/indexnode.go:116] ["New IndexNode ..."]
[2024/04/02 09:07:05.332 +00:00] [DEBUG] [indexnode/service.go:87] [IndexNode] ["network address"=172.21.0.5:21121] ["network port: "=21121]
[2024/04/02 09:07:05.333 +00:00] [INFO] [proxy/lb_policy.go:78] ["use look_aside policy on replica selection"]
[2024/04/02 09:07:05.333 +00:00] [INFO] [runtime/asm_amd64.s:1598] ["Start check query node health loop"]
[2024/04/02 09:07:05.333 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[etcd:2379]"] [minVersion=1.3]
[2024/04/02 09:07:05.334 +00:00] [DEBUG] [querycoord/service.go:215] [network] [port=19531]

The end:

[2024/04/16 15:12:11.428 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
[2024/04/16 15:12:11.831 +00:00] [INFO] [datacoord/meta.go:1324] ["UpdateChannelCheckpoint done"] [vChannel=by-dev-rootcoord-dml_3_444725728207049774v0] [ts=449126159240134657] [msgID="\ufffd\ufffdsjgx:\u0006"] [time=2024/04/16 15:12:11.574 +00:00]
[2024/04/16 15:12:11.834 +00:00] [INFO] [datacoord/meta.go:1324] ["UpdateChannelCheckpoint done"] [vChannel=by-dev-rootcoord-dml_0_447880825398106991v0] [ts=449126111372640257] [msgID=")\ufffdsjgx:\u0006"] [time=2024/04/16 15:09:08.974 +00:00]
[2024/04/16 15:12:11.835 +00:00] [INFO] [datacoord/meta.go:1324] ["UpdateChannelCheckpoint done"] [vChannel=by-dev-rootcoord-dml_2_444725728207049736v0] [ts=449126159240134657] [msgID="\ufffd\ufffdsjgx:\u0006"] [time=2024/04/16 15:12:11.574 +00:00]
[2024/04/16 15:12:11.837 +00:00] [INFO] [datacoord/meta.go:1324] ["UpdateChannelCheckpoint done"] [vChannel=by-dev-rootcoord-dml_6_444725728210909993v0] [ts=449126159240134657] [msgID="\ufffd\ufffdsjgx:\u0006"] [time=2024/04/16 15:12:11.574 +00:00]
[2024/04/16 15:12:12.429 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
[2024/04/16 15:12:12.911 +00:00] [INFO] [proxy/meta_cache.go:900] ["expire all shard leader cache"] [database=default] [collections="[vectors_ml_e5_large,ai_helpdesk,documents_EN,documents_DE,documents,document_texts]"]
[2024/04/16 15:12:13.428 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
[2024/04/16 15:12:14.434 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
[2024/04/16 15:12:15.429 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
[2024/04/16 15:12:15.649 +00:00] [INFO] [datacoord/index_service.go:513] ["receive DescribeIndex request"] [traceID=f19bbbd5696a39f07e531de444475694] [collectionID=447880825398107025] [indexName=] [timestamp=0]
[2024/04/16 15:12:15.650 +00:00] [INFO] [datacoord/index_service.go:513] ["receive DescribeIndex request"] [traceID=8838bd562f709815d9aa5a4865043ca6] [collectionID=444725728210909993] [indexName=] [timestamp=0]
[2024/04/16 15:12:15.650 +00:00] [INFO] [datacoord/index_service.go:449] ["completeIndexInfo success"] [collectionID=444725728210909993] [indexID=447042714307813983] [totalRows=530271] [indexRows=530271] [pendingIndexRows=0] [state=Finished] [failReason=]
[2024/04/16 15:12:15.651 +00:00] [INFO] [datacoord/index_service.go:560] ["DescribeIndex success"] [traceID=8838bd562f709815d9aa5a4865043ca6] [collectionID=444725728210909993] [indexName=]
[2024/04/16 15:12:15.650 +00:00] [INFO] [datacoord/index_service.go:449] ["completeIndexInfo success"] [collectionID=447880825398107025] [indexID=447880825398107030] [totalRows=272665] [indexRows=272665] [pendingIndexRows=129433] [state=Finished] [failReason=]
[2024/04/16 15:12:15.651 +00:00] [INFO] [datacoord/index_service.go:560] ["DescribeIndex success"] [traceID=f19bbbd5696a39f07e531de444475694] [collectionID=447880825398107025] [indexName=]
[2024/04/16 15:12:15.650 +00:00] [INFO] [datacoord/index_service.go:513] ["receive DescribeIndex request"] [traceID=5c5ef398d78f7715ad1d82b5626d3bdc] [collectionID=444725728207049736] [indexName=] [timestamp=0]
[2024/04/16 15:12:15.652 +00:00] [INFO] [datacoord/index_service.go:449] ["completeIndexInfo success"] [collectionID=444725728207049736] [indexID=444725728207049740] [totalRows=9069] [indexRows=9069] [pendingIndexRows=0] [state=Finished] [failReason=]
[2024/04/16 15:12:15.652 +00:00] [INFO] [datacoord/index_service.go:560] ["DescribeIndex success"] [traceID=5c5ef398d78f7715ad1d82b5626d3bdc] [collectionID=444725728207049736] [indexName=]
[2024/04/16 15:12:15.654 +00:00] [INFO] [querynodev2/services.go:1366] ["sync action"] [traceID=506701dfa13daae0cf8e988850b7e0f6] [collectionID=444725728210909993] [channel=by-dev-rootcoord-dml_6_444725728210909993v0] [currentNodeID=12] [Action=UpdateVersion] [TargetVersion=1713280325656191319]
[2024/04/16 15:12:15.650 +00:00] [INFO] [datacoord/index_service.go:513] ["receive DescribeIndex request"] [traceID=646294324f36ec35e92a16d8ad6f6190] [collectionID=447042714314267305] [indexName=] [timestamp=0]
[2024/04/16 15:12:15.650 +00:00] [INFO] [datacoord/index_service.go:513] ["receive DescribeIndex request"] [traceID=ffc76ddf1363520ca0595062c1696199] [collectionID=447880825398106991] [indexName=] [timestamp=0]
[2024/04/16 15:12:15.650 +00:00] [INFO] [datacoord/index_service.go:513] ["receive DescribeIndex request"] [traceID=63ec49683d8af69062e072cb2e9880b5] [collectionID=444725728207049774] [indexName=] [timestamp=0]
[2024/04/16 15:12:15.655 +00:00] [INFO] [querynodev2/services.go:1366] ["sync action"] [traceID=5bfb7260f6cae5528f1f2e3ef0e329df] [collectionID=444725728207049736] [channel=by-dev-rootcoord-dml_2_444725728207049736v0] [currentNodeID=12] [Action=UpdateVersion] [TargetVersion=1713280325658861540]
[2024/04/16 15:12:15.655 +00:00] [INFO] [querynodev2/services.go:1366] ["sync action"] [traceID=3f6befb4928d5e8e44932a3ef7ce9ae5] [collectionID=447880825398107025] [channel=by-dev-rootcoord-dml_1_447880825398107025v0] [currentNodeID=12] [Action=UpdateVersion] [TargetVersion=1713280325654978717]
[2024/04/16 15:12:15.657 +00:00] [INFO] [delegator/distribution.go:294] ["Update readable segment version"] [oldVersion=1713280315653425196] [newVersion=1713280325654978717] [growingSegmentNum=0] [sealedSegmentNum=3]
[2024/04/16 15:12:15.657 +00:00] [INFO] [delegator/distribution.go:294] ["Update readable segment version"] [oldVersion=1713280315662337894] [newVersion=1713280325656191319] [growingSegmentNum=0] [sealedSegmentNum=6]
[2024/04/16 15:12:15.658 +00:00] [INFO] [datacoord/index_service.go:449] ["completeIndexInfo success"] [collectionID=444725728207049774] [indexID=444725728207049778] [totalRows=9069] [indexRows=9069] [pendingIndexRows=0] [state=Finished] [failReason=]
[2024/04/16 15:12:15.658 +00:00] [INFO] [datacoord/index_service.go:560] ["DescribeIndex success"] [traceID=63ec49683d8af69062e072cb2e9880b5] [collectionID=444725728207049774] [indexName=]
[2024/04/16 15:12:15.658 +00:00] [INFO] [datacoord/index_service.go:449] ["completeIndexInfo success"] [collectionID=447042714314267305] [indexID=447042714314267311] [totalRows=35] [indexRows=35] [pendingIndexRows=0] [state=Finished] [failReason=]
[2024/04/16 15:12:15.658 +00:00] [INFO] [datacoord/index_service.go:560] ["DescribeIndex success"] [traceID=646294324f36ec35e92a16d8ad6f6190] [collectionID=447042714314267305] [indexName=]
[2024/04/16 15:12:15.658 +00:00] [INFO] [datacoord/index_service.go:449] ["completeIndexInfo success"] [collectionID=447880825398106991] [indexID=447880825398106997] [totalRows=271127] [indexRows=271127] [pendingIndexRows=0] [state=Finished] [failReason=]
[2024/04/16 15:12:15.659 +00:00] [INFO] [datacoord/index_service.go:560] ["DescribeIndex success"] [traceID=ffc76ddf1363520ca0595062c1696199] [collectionID=447880825398106991] [indexName=]
[2024/04/16 15:12:15.658 +00:00] [INFO] [delegator/distribution.go:294] ["Update readable segment version"] [oldVersion=1713280315664583265] [newVersion=1713280325658861540] [growingSegmentNum=0] [sealedSegmentNum=2]
[2024/04/16 15:12:15.659 +00:00] [INFO] [observers/target_observer.go:473] ["observer trigger update current target"] [collectionID=444725728207049736]
[2024/04/16 15:12:15.660 +00:00] [INFO] [datacoord/services.go:757] ["get recovery info request received"] [traceID=eb1fa7d81d3e30624dd0260d48dda8fe] [collectionID=447880825398107025] [partitionIDs="[]"]
[2024/04/16 15:12:15.662 +00:00] [INFO] [datacoord/handler.go:116] [GetQueryVChanPositions] [collectionID=447880825398107025] [channel=by-dev-rootcoord-dml_1_447880825398107025v0] [numOfSegments=6] ["indexed segment"=4]
[2024/04/16 15:12:15.661 +00:00] [INFO] [querynodev2/services.go:1366] ["sync action"] [traceID=eec5b9edd984f116febea1cc2dedda55] [collectionID=447042714314267305] [channel=by-dev-rootcoord-dml_7_447042714314267305v0] [currentNodeID=12] [Action=UpdateVersion] [TargetVersion=1713280325659637243]
[2024/04/16 15:12:15.661 +00:00] [INFO] [datacoord/services.go:757] ["get recovery info request received"] [traceID=ca598e81ec98311b57263b6018fdf23b] [collectionID=444725728210909993] [partitionIDs="[]"]
[2024/04/16 15:12:15.662 +00:00] [INFO] [querynodev2/services.go:1366] ["sync action"] [traceID=3fd75d940755f2f29b69186c88913f27] [collectionID=444725728207049774] [channel=by-dev-rootcoord-dml_3_444725728207049774v0] [currentNodeID=12] [Action=UpdateVersion] [TargetVersion=1713280325657230648]
[2024/04/16 15:12:15.663 +00:00] [INFO] [delegator/distribution.go:294] ["Update readable segment version"] [oldVersion=1713280315653937742] [newVersion=1713280325657230648] [growingSegmentNum=0] [sealedSegmentNum=2]
[2024/04/16 15:12:15.663 +00:00] [INFO] [datacoord/handler.go:116] [GetQueryVChanPositions] [collectionID=444725728210909993] [channel=by-dev-rootcoord-dml_6_444725728210909993v0] [numOfSegments=6] ["indexed segment"=6]
[2024/04/16 15:12:15.663 +00:00] [INFO] [datacoord/handler.go:299] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_6_444725728210909993v0] [posTs=449126159240134657] [posTime=2024/04/16 15:12:11.574 +00:00]
[2024/04/16 15:12:15.663 +00:00] [INFO] [datacoord/services.go:772] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=ca598e81ec98311b57263b6018fdf23b] [collectionID=444725728210909993] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_6_444725728210909993v0] ["# of unflushed segments"=0] ["# of flushed segments"=6] ["# of dropped segments"=0] ["# of indexed segments"=0]
[2024/04/16 15:12:15.663 +00:00] [INFO] [datacoord/handler.go:299] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_1_447880825398107025v0] [posTs=449126111372640257] [posTime=2024/04/16 15:09:08.974 +00:00]
[2024/04/16 15:12:15.664 +00:00] [INFO] [datacoord/services.go:772] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=eb1fa7d81d3e30624dd0260d48dda8fe] [collectionID=447880825398107025] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_1_447880825398107025v0] ["# of unflushed segments"=0] ["# of flushed segments"=3] ["# of dropped segments"=1] ["# of indexed segments"=0]
[2024/04/16 15:12:15.662 +00:00] [INFO] [datacoord/services.go:757] ["get recovery info request received"] [traceID=3d36a58f4deb6f100332872f8e8a7a15] [collectionID=444725728207049736] [partitionIDs="[]"]
[2024/04/16 15:12:15.663 +00:00] [INFO] [delegator/distribution.go:294] ["Update readable segment version"] [oldVersion=1713280315661582892] [newVersion=1713280325659637243] [growingSegmentNum=0] [sealedSegmentNum=1]
[2024/04/16 15:12:15.664 +00:00] [INFO] [querynodev2/services.go:1366] ["sync action"] [traceID=e0fda5bf84419d01ec0b1de09e4eca32] [collectionID=447880825398106991] [channel=by-dev-rootcoord-dml_0_447880825398106991v0] [currentNodeID=12] [Action=UpdateVersion] [TargetVersion=1713280325655641871]
[2024/04/16 15:12:15.665 +00:00] [INFO] [delegator/distribution.go:294] ["Update readable segment version"] [oldVersion=1713280315653371467] [newVersion=1713280325655641871] [growingSegmentNum=1] [sealedSegmentNum=1]
[2024/04/16 15:12:15.665 +00:00] [INFO] [datacoord/handler.go:116] [GetQueryVChanPositions] [collectionID=444725728207049736] [channel=by-dev-rootcoord-dml_2_444725728207049736v0] [numOfSegments=2] ["indexed segment"=2]
[2024/04/16 15:12:15.667 +00:00] [INFO] [datacoord/handler.go:299] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_2_444725728207049736v0] [posTs=449126159240134657] [posTime=2024/04/16 15:12:11.574 +00:00]
[2024/04/16 15:12:15.667 +00:00] [INFO] [datacoord/services.go:772] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=3d36a58f4deb6f100332872f8e8a7a15] [collectionID=444725728207049736] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_2_444725728207049736v0] ["# of unflushed segments"=0] ["# of flushed segments"=2] ["# of dropped segments"=0] ["# of indexed segments"=0]
[2024/04/16 15:12:15.666 +00:00] [INFO] [datacoord/services.go:757] ["get recovery info request received"] [traceID=210583f2f932c873de2badbc03bdef73] [collectionID=444725728207049774] [partitionIDs="[]"]
[2024/04/16 15:12:15.667 +00:00] [INFO] [datacoord/handler.go:116] [GetQueryVChanPositions] [collectionID=444725728207049774] [channel=by-dev-rootcoord-dml_3_444725728207049774v0] [numOfSegments=2] ["indexed segment"=2]
[2024/04/16 15:12:15.666 +00:00] [INFO] [datacoord/services.go:757] ["get recovery info request received"] [traceID=679067e6d3640f3166fe4f0bded06a51] [collectionID=447880825398106991] [partitionIDs="[]"]
[2024/04/16 15:12:15.666 +00:00] [INFO] [datacoord/services.go:757] ["get recovery info request received"] [traceID=9b33e06be60de7980727a7c56eee4606] [collectionID=447042714314267305] [partitionIDs="[]"]
[2024/04/16 15:12:15.668 +00:00] [INFO] [datacoord/handler.go:299] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_3_444725728207049774v0] [posTs=449126159240134657] [posTime=2024/04/16 15:12:11.574 +00:00]
[2024/04/16 15:12:15.668 +00:00] [INFO] [datacoord/services.go:772] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=210583f2f932c873de2badbc03bdef73] [collectionID=444725728207049774] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_3_444725728207049774v0] ["# of unflushed segments"=0] ["# of flushed segments"=2] ["# of dropped segments"=0] ["# of indexed segments"=0]
[2024/04/16 15:12:15.668 +00:00] [INFO] [datacoord/handler.go:116] [GetQueryVChanPositions] [collectionID=447880825398106991] [channel=by-dev-rootcoord-dml_0_447880825398106991v0] [numOfSegments=2] ["indexed segment"=1]
[2024/04/16 15:12:15.669 +00:00] [INFO] [datacoord/handler.go:299] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_0_447880825398106991v0] [posTs=449126111372640257] [posTime=2024/04/16 15:09:08.974 +00:00]
[2024/04/16 15:12:15.669 +00:00] [INFO] [datacoord/services.go:772] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=679067e6d3640f3166fe4f0bded06a51] [collectionID=447880825398106991] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_0_447880825398106991v0] ["# of unflushed segments"=1] ["# of flushed segments"=1] ["# of dropped segments"=0] ["# of indexed segments"=0]
[2024/04/16 15:12:15.668 +00:00] [INFO] [datacoord/handler.go:116] [GetQueryVChanPositions] [collectionID=447042714314267305] [channel=by-dev-rootcoord-dml_7_447042714314267305v0] [numOfSegments=1] ["indexed segment"=1]
[2024/04/16 15:12:15.669 +00:00] [INFO] [datacoord/handler.go:299] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_7_447042714314267305v0] [posTs=449126153420537857] [posTime=2024/04/16 15:11:49.374 +00:00]
[2024/04/16 15:12:15.669 +00:00] [INFO] [datacoord/services.go:772] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=9b33e06be60de7980727a7c56eee4606] [collectionID=447042714314267305] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_7_447042714314267305v0] ["# of unflushed segments"=0] ["# of flushed segments"=1] ["# of dropped segments"=0] ["# of indexed segments"=0]
[2024/04/16 15:12:15.912 +00:00] [INFO] [proxy/meta_cache.go:900] ["expire all shard leader cache"] [database=default] [collections="[documents_DE,documents,document_texts,vectors_ml_e5_large,ai_helpdesk,documents_EN]"]
[2024/04/16 15:12:16.428 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
[2024/04/16 15:12:17.428 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
[2024/04/16 15:12:18.428 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
[2024/04/16 15:12:18.910 +00:00] [INFO] [proxy/meta_cache.go:900] ["expire all shard leader cache"] [database=default] [collections="[documents,document_texts,vectors_ml_e5_large,ai_helpdesk,documents_EN,documents_DE]"]
[2024/04/16 15:12:19.433 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
[2024/04/16 15:12:20.431 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
[2024/04/16 15:12:21.429 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
[2024/04/16 15:12:21.911 +00:00] [INFO] [proxy/meta_cache.go:900] ["expire all shard leader cache"] [database=default] [collections="[document_texts,vectors_ml_e5_large,ai_helpdesk,documents_EN,documents_DE,documents]"]
[2024/04/16 15:12:22.428 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
[2024/04/16 15:12:23.432 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
[2024/04/16 15:12:24.429 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
[2024/04/16 15:12:24.913 +00:00] [INFO] [proxy/meta_cache.go:900] ["expire all shard leader cache"] [database=default] [collections="[documents_EN,documents_DE,documents,document_texts,vectors_ml_e5_large,ai_helpdesk]"]
[2024/04/16 15:12:25.428 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
[2024/04/16 15:12:25.649 +00:00] [INFO] [datacoord/index_service.go:513] ["receive DescribeIndex request"] [traceID=f64fc74364475dd268c64da4e4465858] [collectionID=447042714314267305] [indexName=] [timestamp=0]
[2024/04/16 15:12:25.649 +00:00] [INFO] [datacoord/index_service.go:449] ["completeIndexInfo success"] [collectionID=447042714314267305] [indexID=447042714314267311] [totalRows=35] [indexRows=35] [pendingIndexRows=0] [state=Finished] [failReason=]
[2024/04/16 15:12:25.649 +00:00] [INFO] [datacoord/index_service.go:560] ["DescribeIndex success"] [traceID=f64fc74364475dd268c64da4e4465858] [collectionID=447042714314267305] [indexName=]
[2024/04/16 15:12:25.649 +00:00] [INFO] [datacoord/index_service.go:513] ["receive DescribeIndex request"] [traceID=f6f435c6cd6cde94b6d94dd7af0aa1e2] [collectionID=444725728207049736] [indexName=] [timestamp=0]
[2024/04/16 15:12:25.649 +00:00] [INFO] [datacoord/index_service.go:449] ["completeIndexInfo success"] [collectionID=444725728207049736] [indexID=444725728207049740] [totalRows=9069] [indexRows=9069] [pendingIndexRows=0] [state=Finished] [failReason=]
[2024/04/16 15:12:25.649 +00:00] [INFO] [datacoord/index_service.go:560] ["DescribeIndex success"] [traceID=f6f435c6cd6cde94b6d94dd7af0aa1e2] [collectionID=444725728207049736] [indexName=]
[2024/04/16 15:12:25.649 +00:00] [INFO] [datacoord/index_service.go:513] ["receive DescribeIndex request"] [traceID=03cc858ae36eda8b4a02c1563029d716] [collectionID=444725728210909993] [indexName=] [timestamp=0]
[2024/04/16 15:12:25.649 +00:00] [INFO] [datacoord/index_service.go:449] ["completeIndexInfo success"] [collectionID=444725728210909993] [indexID=447042714307813983] [totalRows=530271] [indexRows=530271] [pendingIndexRows=0] [state=Finished] [failReason=]
[2024/04/16 15:12:25.649 +00:00] [INFO] [datacoord/index_service.go:560] ["DescribeIndex success"] [traceID=03cc858ae36eda8b4a02c1563029d716] [collectionID=444725728210909993] [indexName=]
[2024/04/16 15:12:25.650 +00:00] [INFO] [datacoord/index_service.go:513] ["receive DescribeIndex request"] [traceID=11f3315add47168bb6face6c9b059aea] [collectionID=444725728207049774] [indexName=] [timestamp=0]
[2024/04/16 15:12:25.650 +00:00] [INFO] [datacoord/index_service.go:449] ["completeIndexInfo success"] [collectionID=444725728207049774] [indexID=444725728207049778] [totalRows=9069] [indexRows=9069] [pendingIndexRows=0] [state=Finished] [failReason=]
[2024/04/16 15:12:25.650 +00:00] [INFO] [datacoord/index_service.go:560] ["DescribeIndex success"] [traceID=11f3315add47168bb6face6c9b059aea] [collectionID=444725728207049774] [indexName=]
[2024/04/16 15:12:25.650 +00:00] [INFO] [querynodev2/services.go:1366] ["sync action"] [traceID=1d5c6e96807061f1e249897bf0dbb9fb] [collectionID=447042714314267305] [channel=by-dev-rootcoord-dml_7_447042714314267305v0] [currentNodeID=12] [Action=UpdateVersion] [TargetVersion=1713280335666048447]
[2024/04/16 15:12:25.650 +00:00] [INFO] [delegator/distribution.go:294] ["Update readable segment version"] [oldVersion=1713280325659637243] [newVersion=1713280335666048447] [growingSegmentNum=0] [sealedSegmentNum=1]
[2024/04/16 15:12:25.650 +00:00] [INFO] [datacoord/index_service.go:513] ["receive DescribeIndex request"] [traceID=79ea0048efa7b86ccaea8cf6bf7e6a40] [collectionID=447880825398106991] [indexName=] [timestamp=0]
[2024/04/16 15:12:25.651 +00:00] [INFO] [datacoord/index_service.go:449] ["completeIndexInfo success"] [collectionID=447880825398106991] [indexID=447880825398106997] [totalRows=271127] [indexRows=271127] [pendingIndexRows=0] [state=Finished] [failReason=]
[2024/04/16 15:12:25.651 +00:00] [INFO] [datacoord/index_service.go:560] ["DescribeIndex success"] [traceID=79ea0048efa7b86ccaea8cf6bf7e6a40] [collectionID=447880825398106991] [indexName=]
[2024/04/16 15:12:25.650 +00:00] [INFO] [datacoord/index_service.go:513] ["receive DescribeIndex request"] [traceID=6da94005a6f650a753c4380f20d055e3] [collectionID=447880825398107025] [indexName=] [timestamp=0]
[2024/04/16 15:12:25.651 +00:00] [INFO] [datacoord/index_service.go:449] ["completeIndexInfo success"] [collectionID=447880825398107025] [indexID=447880825398107030] [totalRows=272665] [indexRows=272665] [pendingIndexRows=129433] [state=Finished] [failReason=]
[2024/04/16 15:12:25.651 +00:00] [INFO] [datacoord/index_service.go:560] ["DescribeIndex success"] [traceID=6da94005a6f650a753c4380f20d055e3] [collectionID=447880825398107025] [indexName=]
[2024/04/16 15:12:25.651 +00:00] [INFO] [querynodev2/services.go:1366] ["sync action"] [traceID=94f9eb439f1c52a27eef0b77be8da23a] [collectionID=444725728207049736] [channel=by-dev-rootcoord-dml_2_444725728207049736v0] [currentNodeID=12] [Action=UpdateVersion] [TargetVersion=1713280335661490246]
[2024/04/16 15:12:25.651 +00:00] [INFO] [delegator/distribution.go:294] ["Update readable segment version"] [oldVersion=1713280325658861540] [newVersion=1713280335661490246] [growingSegmentNum=0] [sealedSegmentNum=2]
[2024/04/16 15:12:25.651 +00:00] [INFO] [querynodev2/services.go:1366] ["sync action"] [traceID=c0adadb3ce6c1ef0fcf5658486aa9141] [collectionID=444725728207049774] [channel=by-dev-rootcoord-dml_3_444725728207049774v0] [currentNodeID=12] [Action=UpdateVersion] [TargetVersion=1713280335664399979]
[2024/04/16 15:12:25.652 +00:00] [INFO] [delegator/distribution.go:294] ["Update readable segment version"] [oldVersion=1713280325657230648] [newVersion=1713280335664399979] [growingSegmentNum=0] [sealedSegmentNum=2]
[2024/04/16 15:12:25.652 +00:00] [INFO] [querynodev2/services.go:1366] ["sync action"] [traceID=3dc7c6d4c35f19746db24081e4c3a7ef] [collectionID=444725728210909993] [channel=by-dev-rootcoord-dml_6_444725728210909993v0] [currentNodeID=12] [Action=UpdateVersion] [TargetVersion=1713280335661133232]
[2024/04/16 15:12:25.653 +00:00] [INFO] [delegator/distribution.go:294] ["Update readable segment version"] [oldVersion=1713280325656191319] [newVersion=1713280335661133232] [growingSegmentNum=0] [sealedSegmentNum=6]
[2024/04/16 15:12:25.653 +00:00] [INFO] [querynodev2/services.go:1366] ["sync action"] [traceID=a4356cfe07ba0d0c89131c02db36e757] [collectionID=447880825398106991] [channel=by-dev-rootcoord-dml_0_447880825398106991v0] [currentNodeID=12] [Action=UpdateVersion] [TargetVersion=1713280335666636444]
[2024/04/16 15:12:25.653 +00:00] [INFO] [delegator/distribution.go:294] ["Update readable segment version"] [oldVersion=1713280325655641871] [newVersion=1713280335666636444] [growingSegmentNum=1] [sealedSegmentNum=1]
[2024/04/16 15:12:25.653 +00:00] [INFO] [observers/target_observer.go:473] ["observer trigger update current target"] [collectionID=447042714314267305]
[2024/04/16 15:12:25.653 +00:00] [INFO] [querynodev2/services.go:1366] ["sync action"] [traceID=d81fc6efd74998763e728f64ed1fed6e] [collectionID=447880825398107025] [channel=by-dev-rootcoord-dml_1_447880825398107025v0] [currentNodeID=12] [Action=UpdateVersion] [TargetVersion=1713280335660187976]
[2024/04/16 15:12:25.654 +00:00] [INFO] [delegator/distribution.go:294] ["Update readable segment version"] [oldVersion=1713280325654978717] [newVersion=1713280335660187976] [growingSegmentNum=0] [sealedSegmentNum=3]
[2024/04/16 15:12:25.653 +00:00] [INFO] [datacoord/services.go:757] ["get recovery info request received"] [traceID=7ed73cc39f75d9f12e3de83c340f420c] [collectionID=447042714314267305] [partitionIDs="[]"]
[2024/04/16 15:12:25.654 +00:00] [INFO] [datacoord/handler.go:116] [GetQueryVChanPositions] [collectionID=447042714314267305] [channel=by-dev-rootcoord-dml_7_447042714314267305v0] [numOfSegments=1] ["indexed segment"=1]
[2024/04/16 15:12:25.654 +00:00] [INFO] [datacoord/handler.go:299] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_7_447042714314267305v0] [posTs=449126153420537857] [posTime=2024/04/16 15:11:49.374 +00:00]
[2024/04/16 15:12:25.655 +00:00] [INFO] [datacoord/services.go:772] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=7ed73cc39f75d9f12e3de83c340f420c] [collectionID=447042714314267305] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_7_447042714314267305v0] ["# of unflushed segments"=0] ["# of flushed segments"=1] ["# of dropped segments"=0] ["# of indexed segments"=0]
[2024/04/16 15:12:25.654 +00:00] [INFO] [datacoord/services.go:757] ["get recovery info request received"] [traceID=791315521ffdd06f0ad1a21d07fe3cdf] [collectionID=444725728210909993] [partitionIDs="[]"]
[2024/04/16 15:12:25.656 +00:00] [INFO] [datacoord/handler.go:116] [GetQueryVChanPositions] [collectionID=444725728210909993] [channel=by-dev-rootcoord-dml_6_444725728210909993v0] [numOfSegments=6] ["indexed segment"=6]
[2024/04/16 15:12:25.656 +00:00] [INFO] [datacoord/handler.go:299] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_6_444725728210909993v0] [posTs=449126159240134657] [posTime=2024/04/16 15:12:11.574 +00:00]
[2024/04/16 15:12:25.656 +00:00] [INFO] [datacoord/services.go:772] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=791315521ffdd06f0ad1a21d07fe3cdf] [collectionID=444725728210909993] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_6_444725728210909993v0] ["# of unflushed segments"=0] ["# of flushed segments"=6] ["# of dropped segments"=0] ["# of indexed segments"=0]
[2024/04/16 15:12:25.655 +00:00] [INFO] [datacoord/services.go:757] ["get recovery info request received"] [traceID=d44e88f220f5e73cbb375a9e15af364c] [collectionID=447880825398106991] [partitionIDs="[]"]
[2024/04/16 15:12:25.657 +00:00] [INFO] [datacoord/handler.go:116] [GetQueryVChanPositions] [collectionID=447880825398106991] [channel=by-dev-rootcoord-dml_0_447880825398106991v0] [numOfSegments=2] ["indexed segment"=1]
[2024/04/16 15:12:25.657 +00:00] [INFO] [datacoord/handler.go:299] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_0_447880825398106991v0] [posTs=449126111372640257] [posTime=2024/04/16 15:09:08.974 +00:00]
[2024/04/16 15:12:25.657 +00:00] [INFO] [datacoord/services.go:772] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=d44e88f220f5e73cbb375a9e15af364c] [collectionID=447880825398106991] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_0_447880825398106991v0] ["# of unflushed segments"=1] ["# of flushed segments"=1] ["# of dropped segments"=0] ["# of indexed segments"=0]
[2024/04/16 15:12:25.655 +00:00] [INFO] [datacoord/services.go:757] ["get recovery info request received"] [traceID=3f8eb5c2e9571f887956e83162541ebe] [collectionID=444725728207049736] [partitionIDs="[]"]
[2024/04/16 15:12:25.658 +00:00] [INFO] [datacoord/handler.go:116] [GetQueryVChanPositions] [collectionID=444725728207049736] [channel=by-dev-rootcoord-dml_2_444725728207049736v0] [numOfSegments=2] ["indexed segment"=2]
[2024/04/16 15:12:25.658 +00:00] [INFO] [datacoord/handler.go:299] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_2_444725728207049736v0] [posTs=449126159240134657] [posTime=2024/04/16 15:12:11.574 +00:00]
[2024/04/16 15:12:25.655 +00:00] [INFO] [datacoord/services.go:757] ["get recovery info request received"] [traceID=68f4bc48d2c9d4a703d313b5b7535c7f] [collectionID=444725728207049774] [partitionIDs="[]"]
[2024/04/16 15:12:25.655 +00:00] [INFO] [datacoord/services.go:757] ["get recovery info request received"] [traceID=34f21350a04ed9602e8470e4d524fe8f] [collectionID=447880825398107025] [partitionIDs="[]"]
[2024/04/16 15:12:25.658 +00:00] [INFO] [datacoord/handler.go:116] [GetQueryVChanPositions] [collectionID=447880825398107025] [channel=by-dev-rootcoord-dml_1_447880825398107025v0] [numOfSegments=6] ["indexed segment"=4]
[2024/04/16 15:12:25.659 +00:00] [INFO] [datacoord/handler.go:299] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_1_447880825398107025v0] [posTs=449126111372640257] [posTime=2024/04/16 15:09:08.974 +00:00]
[2024/04/16 15:12:25.659 +00:00] [INFO] [datacoord/services.go:772] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=34f21350a04ed9602e8470e4d524fe8f] [collectionID=447880825398107025] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_1_447880825398107025v0] ["# of unflushed segments"=0] ["# of flushed segments"=3] ["# of dropped segments"=1] ["# of indexed segments"=0]
[2024/04/16 15:12:25.659 +00:00] [INFO] [datacoord/services.go:772] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=3f8eb5c2e9571f887956e83162541ebe] [collectionID=444725728207049736] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_2_444725728207049736v0] ["# of unflushed segments"=0] ["# of flushed segments"=2] ["# of dropped segments"=0] ["# of indexed segments"=0]
[2024/04/16 15:12:25.659 +00:00] [INFO] [datacoord/handler.go:116] [GetQueryVChanPositions] [collectionID=444725728207049774] [channel=by-dev-rootcoord-dml_3_444725728207049774v0] [numOfSegments=2] ["indexed segment"=2]
[2024/04/16 15:12:25.660 +00:00] [INFO] [datacoord/handler.go:299] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_3_444725728207049774v0] [posTs=449126159240134657] [posTime=2024/04/16 15:12:11.574 +00:00]
[2024/04/16 15:12:25.660 +00:00] [INFO] [datacoord/services.go:772] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=68f4bc48d2c9d4a703d313b5b7535c7f] [collectionID=444725728207049774] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_3_444725728207049774v0] ["# of unflushed segments"=0] ["# of flushed segments"=2] ["# of dropped segments"=0] ["# of indexed segments"=0]
[2024/04/16 15:12:26.429 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
[2024/04/16 15:12:27.429 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
[2024/04/16 15:12:27.697 +00:00] [INFO] [proxy/meta_cache.go:802] ["no shard cache for collection, try to get shard leaders from QueryCoord"] [traceID=7cf7a80fcc0ab91d8032a48f90ee07fa] [collectionName=document_texts] [collectionID=447880825398106991]
[2024/04/16 15:12:27.705 +00:00] [INFO] [datacoord/services.go:174] ["handle assign segment request"] [traceID=caecc67a21e3fab2e6fa9112cc4a2bc2] [collectionID=447880825398106991] [partitionID=447880825398106992] [channelName=by-dev-rootcoord-dml_0_447880825398106991v0] [count=7] [isImport=false] ["import task ID"=0]
[2024/04/16 15:12:27.706 +00:00] [INFO] [datacoord/meta.go:938] ["meta update: add allocation - complete"] [segmentID=448803498411568777]
[2024/04/16 15:12:27.706 +00:00] [INFO] [datacoord/services.go:213] ["success to assign segments"] [traceID=caecc67a21e3fab2e6fa9112cc4a2bc2] [collectionID=447880825398106991] [assignments="[{\"SegmentID\":448803498411568777,\"NumOfRows\":7,\"ExpireTime\":449126163984941059}]"]
[2024/04/16 15:12:27.712 +00:00] [INFO] [datacoord/services.go:174] ["handle assign segment request"] [traceID=afd43dc810910dcf4a01f398b656ee7d] [collectionID=447880825398107025] [partitionID=447880825398107026] [channelName=by-dev-rootcoord-dml_1_447880825398107025v0] [count=7] [isImport=false] ["import task ID"=0]
[2024/04/16 15:12:27.713 +00:00] [INFO] [datacoord/meta.go:938] ["meta update: add allocation - complete"] [segmentID=448803498411599794]
[2024/04/16 15:12:27.713 +00:00] [INFO] [datacoord/services.go:213] ["success to assign segments"] [traceID=afd43dc810910dcf4a01f398b656ee7d] [collectionID=447880825398107025] [assignments="[{\"SegmentID\":448803498411599794,\"NumOfRows\":7,\"ExpireTime\":449126163984941061}]"]
[2024/04/16 15:12:27.911 +00:00] [INFO] [proxy/meta_cache.go:900] ["expire all shard leader cache"] [database=default] [collections="[documents,document_texts,vectors_ml_e5_large,ai_helpdesk,documents_EN,documents_DE]"]
[2024/04/16 15:12:28.026 +00:00] [INFO] [datanode/channel_meta.go:608] ["updated segment num row"] [segmentID=448803498411599794] [addedNumRows=7] [numRowsSum=58]
[2024/04/16 15:12:28.030 +00:00] [INFO] [datanode/channel_meta.go:634] ["updated segment memorySize"] [segmentID=448803498411599794] [memorySize=249230]
[2024/04/16 15:12:28.030 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:286] ["segment buffer status"] [no.=0] [segmentID=448803498411599794] [channel=by-dev-rootcoord-dml_1_447880825398107025v0] [size=58] [limit=3592] [memorySize=249230]
[2024/04/16 15:12:28.029 +00:00] [INFO] [datanode/channel_meta.go:608] ["updated segment num row"] [segmentID=448803498411568777] [addedNumRows=7] [numRowsSum=2639]
[2024/04/16 15:12:28.030 +00:00] [INFO] [datanode/channel_meta.go:634] ["updated segment memorySize"] [segmentID=448803498411568777] [memorySize=32565]
[2024/04/16 15:12:28.030 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:286] ["segment buffer status"] [no.=0] [segmentID=448803498411568777] [channel=by-dev-rootcoord-dml_0_447880825398106991v0] [size=58] [limit=20069] [memorySize=32565]
[2024/04/16 15:12:28.029 +00:00] [INFO] [pipeline/insert_node.go:80] ["pipeline fetch insert msg"] [collectionID=447880825398106991] [segmentID=448803498411568777] [insertRowNum=7] [timestampMin=449126163460653058] [timestampMax=449126163460653058]
[2024/04/16 15:12:28.029 +00:00] [INFO] [pipeline/insert_node.go:80] ["pipeline fetch insert msg"] [collectionID=447880825398107025] [segmentID=448803498411599794] [insertRowNum=7] [timestampMin=449126163460653060] [timestampMax=449126163460653060]
[2024/04/16 15:12:28.433 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
[2024/04/16 15:12:29.429 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
[2024/04/16 15:12:30.428 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
[2024/04/16 15:12:30.911 +00:00] [INFO] [proxy/meta_cache.go:900] ["expire all shard leader cache"] [database=default] [collections="[document_texts,vectors_ml_e5_large,ai_helpdesk,documents_EN,documents_DE,documents]"]
[2024/04/16 15:12:31.429 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
[2024/04/16 15:12:32.428 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
I20240416 15:12:34.061237 44042 time_recorder.cc:49] [KNOWHERE][PrintTimeRecord][milvus] Build index: done (68980.911921 ms)
[2024/04/16 15:12:34.126 +00:00] [WARN] [tso/tso.go:178] ["clock offset is huge, check network latency and clock skew"] [jet-lag=409.380249ms] [prev-physical=2024/04/16 15:12:33.550 +00:00] [now=2024/04/16 15:12:33.959 +00:00]
[2024/04/16 15:12:34.064 +00:00] [INFO] [proxy/meta_cache.go:900] ["expire all shard leader cache"] [database=default] [collections="[vectors_ml_e5_large,ai_helpdesk,documents_EN,documents_DE,documents,document_texts]"]
[2024/04/16 15:12:33.529 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
[2024/04/16 15:12:34.317 +00:00] [INFO] [indexnode/task.go:363] ["Successfully build index"] [buildID=448803498411600556] [Collection=447880825398107025] [SegmentID=448803498404092843] [currentIndexVersion=1]
I20240416 15:12:34.334169 44059 ivf.cc:699] [KNOWHERE][Serialize][milvus] request version 1
[2024/04/16 15:12:34.412 +00:00] [WARN] [rootcoord/timeticksync.go:313] ["rootcoord send tt to all channels too slowly"] [chanNum=6] [span=213]
[2024/04/16 15:12:34.431 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
I20240416 15:12:35.336944 44059 ivf.cc:705] [KNOWHERE][Serialize][milvus] write IVF_FLAT, file size 535913399
[2024/04/16 15:12:35.429 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
[2024/04/16 15:12:35.650 +00:00] [INFO] [datacoord/index_service.go:513] ["receive DescribeIndex request"] [traceID=a6ed6e8fe9d595751413ca3f6f2c66c9] [collectionID=447880825398106991] [indexName=] [timestamp=0]
[2024/04/16 15:12:35.651 +00:00] [INFO] [datacoord/index_service.go:449] ["completeIndexInfo success"] [collectionID=447880825398106991] [indexID=447880825398106997] [totalRows=271127] [indexRows=271127] [pendingIndexRows=0] [state=Finished] [failReason=]
[2024/04/16 15:12:35.651 +00:00] [INFO] [datacoord/index_service.go:560] ["DescribeIndex success"] [traceID=a6ed6e8fe9d595751413ca3f6f2c66c9] [collectionID=447880825398106991] [indexName=]
[2024/04/16 15:12:35.651 +00:00] [INFO] [datacoord/index_service.go:513] ["receive DescribeIndex request"] [traceID=8db95ba8a613a68680eb1e4b639b0265] [collectionID=444725728207049774] [indexName=] [timestamp=0]
[2024/04/16 15:12:35.651 +00:00] [INFO] [datacoord/index_service.go:449] ["completeIndexInfo success"] [collectionID=444725728207049774] [indexID=444725728207049778] [totalRows=9069] [indexRows=9069] [pendingIndexRows=0] [state=Finished] [failReason=]
[2024/04/16 15:12:35.651 +00:00] [INFO] [datacoord/index_service.go:560] ["DescribeIndex success"] [traceID=8db95ba8a613a68680eb1e4b639b0265] [collectionID=444725728207049774] [indexName=]
[2024/04/16 15:12:35.652 +00:00] [INFO] [datacoord/index_service.go:513] ["receive DescribeIndex request"] [traceID=e87b526eae44760460084b5d020edbd5] [collectionID=444725728210909993] [indexName=] [timestamp=0]
[2024/04/16 15:12:35.652 +00:00] [INFO] [datacoord/index_service.go:449] ["completeIndexInfo success"] [collectionID=444725728210909993] [indexID=447042714307813983] [totalRows=530271] [indexRows=530271] [pendingIndexRows=0] [state=Finished] [failReason=]
[2024/04/16 15:12:35.652 +00:00] [INFO] [datacoord/index_service.go:560] ["DescribeIndex success"] [traceID=e87b526eae44760460084b5d020edbd5] [collectionID=444725728210909993] [indexName=]
[2024/04/16 15:12:35.653 +00:00] [INFO] [querynodev2/services.go:1366] ["sync action"] [traceID=ce3f5e08d8dc35a64a21fb7e269ceb7a] [collectionID=447880825398106991] [channel=by-dev-rootcoord-dml_0_447880825398106991v0] [currentNodeID=12] [Action=UpdateVersion] [TargetVersion=1713280345654142257]
[2024/04/16 15:12:35.656 +00:00] [INFO] [datacoord/index_service.go:513] ["receive DescribeIndex request"] [traceID=8357fa31384ee475a16d696f970ba8fa] [collectionID=447880825398107025] [indexName=] [timestamp=0]
[2024/04/16 15:12:35.656 +00:00] [INFO] [datacoord/index_service.go:449] ["completeIndexInfo success"] [collectionID=447880825398107025] [indexID=447880825398107030] [totalRows=272665] [indexRows=272665] [pendingIndexRows=129433] [state=Finished] [failReason=]
[2024/04/16 15:12:35.656 +00:00] [INFO] [datacoord/index_service.go:560] ["DescribeIndex success"] [traceID=8357fa31384ee475a16d696f970ba8fa] [collectionID=447880825398107025] [indexName=]
[2024/04/16 15:12:35.657 +00:00] [INFO] [datacoord/index_service.go:513] ["receive DescribeIndex request"] [traceID=7ddd149a134895145d1b5b516d11889b] [collectionID=444725728207049736] [indexName=] [timestamp=0]
[2024/04/16 15:12:35.657 +00:00] [INFO] [datacoord/index_service.go:449] ["completeIndexInfo success"] [collectionID=444725728207049736] [indexID=444725728207049740] [totalRows=9069] [indexRows=9069] [pendingIndexRows=0] [state=Finished] [failReason=]
[2024/04/16 15:12:35.657 +00:00] [INFO] [datacoord/index_service.go:560] ["DescribeIndex success"] [traceID=7ddd149a134895145d1b5b516d11889b] [collectionID=444725728207049736] [indexName=]
[2024/04/16 15:12:35.658 +00:00] [INFO] [delegator/distribution.go:294] ["Update readable segment version"] [oldVersion=1713280335666636444] [newVersion=1713280345654142257] [growingSegmentNum=1] [sealedSegmentNum=1]
[2024/04/16 15:12:35.659 +00:00] [INFO] [querynodev2/services.go:1366] ["sync action"] [traceID=dd2a282ee7fd9f9935905aa6df15e23c] [collectionID=444725728210909993] [channel=by-dev-rootcoord-dml_6_444725728210909993v0] [currentNodeID=12] [Action=UpdateVersion] [TargetVersion=1713280345654482140]
[2024/04/16 15:12:35.659 +00:00] [INFO] [delegator/distribution.go:294] ["Update readable segment version"] [oldVersion=1713280335661133232] [newVersion=1713280345654482140] [growingSegmentNum=0] [sealedSegmentNum=6]
[2024/04/16 15:12:35.659 +00:00] [INFO] [observers/target_observer.go:473] ["observer trigger update current target"] [collectionID=444725728210909993]
[2024/04/16 15:12:35.661 +00:00] [INFO] [querynodev2/services.go:1366] ["sync action"] [traceID=687ef4159ab08306e47534eb5878deb4] [collectionID=444725728207049774] [channel=by-dev-rootcoord-dml_3_444725728207049774v0] [currentNodeID=12] [Action=UpdateVersion] [TargetVersion=1713280345654410662]
[2024/04/16 15:12:35.661 +00:00] [INFO] [delegator/distribution.go:294] ["Update readable segment version"] [oldVersion=1713280335664399979] [newVersion=1713280345654410662] [growingSegmentNum=0] [sealedSegmentNum=2]
[2024/04/16 15:12:35.661 +00:00] [INFO] [querynodev2/services.go:1366] ["sync action"] [traceID=52ced16f6607061ba16479c8d10c9287] [collectionID=444725728207049736] [channel=by-dev-rootcoord-dml_2_444725728207049736v0] [currentNodeID=12] [Action=UpdateVersion] [TargetVersion=1713280345654270124]
[2024/04/16 15:12:35.661 +00:00] [INFO] [delegator/distribution.go:294] ["Update readable segment version"] [oldVersion=1713280335661490246] [newVersion=1713280345654270124] [growingSegmentNum=0] [sealedSegmentNum=2]
[2024/04/16 15:12:35.662 +00:00] [INFO] [querynodev2/services.go:1366] ["sync action"] [traceID=82f8572ea738a53dbf84d36d12d47ac2] [collectionID=447880825398107025] [channel=by-dev-rootcoord-dml_1_447880825398107025v0] [currentNodeID=12] [Action=UpdateVersion] [TargetVersion=1713280345654902583]
[2024/04/16 15:12:35.663 +00:00] [INFO] [delegator/distribution.go:294] ["Update readable segment version"] [oldVersion=1713280335660187976] [newVersion=1713280345654902583] [growingSegmentNum=0] [sealedSegmentNum=3]
[2024/04/16 15:12:35.663 +00:00] [INFO] [datacoord/index_service.go:513] ["receive DescribeIndex request"] [traceID=3cb69afe91255ec8fa12b952378caa84] [collectionID=447042714314267305] [indexName=] [timestamp=0]
[2024/04/16 15:12:35.663 +00:00] [INFO] [datacoord/index_service.go:449] ["completeIndexInfo success"] [collectionID=447042714314267305] [indexID=447042714314267311] [totalRows=35] [indexRows=35] [pendingIndexRows=0] [state=Finished] [failReason=]
[2024/04/16 15:12:35.663 +00:00] [INFO] [datacoord/index_service.go:560] ["DescribeIndex success"] [traceID=3cb69afe91255ec8fa12b952378caa84] [collectionID=447042714314267305] [indexName=]
[2024/04/16 15:12:35.663 +00:00] [INFO] [datacoord/services.go:757] ["get recovery info request received"] [traceID=e3333646adb3dd4c4438d5c44e000bb1] [collectionID=447880825398107025] [partitionIDs="[]"]
[2024/04/16 15:12:35.664 +00:00] [INFO] [querynodev2/services.go:1366] ["sync action"] [traceID=c7d0b98e4ce4af35cb93b4e6b4c495fc] [collectionID=447042714314267305] [channel=by-dev-rootcoord-dml_7_447042714314267305v0] [currentNodeID=12] [Action=UpdateVersion] [TargetVersion=1713280345653580932]
[2024/04/16 15:12:35.664 +00:00] [INFO] [delegator/distribution.go:294] ["Update readable segment version"] [oldVersion=1713280335666048447] [newVersion=1713280345653580932] [growingSegmentNum=0] [sealedSegmentNum=1]
[2024/04/16 15:12:35.664 +00:00] [INFO] [datacoord/services.go:757] ["get recovery info request received"] [traceID=0327d0c765063c2bd96f043c29dd8dc2] [collectionID=444725728207049774] [partitionIDs="[]"]
[2024/04/16 15:12:35.664 +00:00] [INFO] [datacoord/services.go:757] ["get recovery info request received"] [traceID=a938811b88cdf6e58940ff02eb78ad3d] [collectionID=447880825398106991] [partitionIDs="[]"]
[2024/04/16 15:12:35.664 +00:00] [INFO] [datacoord/services.go:757] ["get recovery info request received"] [traceID=92a0c28e116f5f68b4550d993bae0417] [collectionID=444725728207049736] [partitionIDs="[]"]
[2024/04/16 15:12:35.665 +00:00] [INFO] [datacoord/handler.go:116] [GetQueryVChanPositions] [collectionID=444725728207049774] [channel=by-dev-rootcoord-dml_3_444725728207049774v0] [numOfSegments=2] ["indexed segment"=2]
[2024/04/16 15:12:35.665 +00:00] [INFO] [datacoord/handler.go:299] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_3_444725728207049774v0] [posTs=449126159240134657] [posTime=2024/04/16 15:12:11.574 +00:00]
[2024/04/16 15:12:35.665 +00:00] [INFO] [datacoord/services.go:772] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=0327d0c765063c2bd96f043c29dd8dc2] [collectionID=444725728207049774] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_3_444725728207049774v0] ["# of unflushed segments"=0] ["# of flushed segments"=2] ["# of dropped segments"=0] ["# of indexed segments"=0]
[2024/04/16 15:12:35.665 +00:00] [INFO] [datacoord/handler.go:116] [GetQueryVChanPositions] [collectionID=444725728207049736] [channel=by-dev-rootcoord-dml_2_444725728207049736v0] [numOfSegments=2] ["indexed segment"=2]
[2024/04/16 15:12:35.665 +00:00] [INFO] [datacoord/handler.go:299] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_2_444725728207049736v0] [posTs=449126159240134657] [posTime=2024/04/16 15:12:11.574 +00:00]
[2024/04/16 15:12:35.665 +00:00] [INFO] [datacoord/services.go:772] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=92a0c28e116f5f68b4550d993bae0417] [collectionID=444725728207049736] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_2_444725728207049736v0] ["# of unflushed segments"=0] ["# of flushed segments"=2] ["# of dropped segments"=0] ["# of indexed segments"=0]
[2024/04/16 15:12:35.665 +00:00] [INFO] [datacoord/handler.go:116] [GetQueryVChanPositions] [collectionID=447880825398107025] [channel=by-dev-rootcoord-dml_1_447880825398107025v0] [numOfSegments=6] ["indexed segment"=4]
[2024/04/16 15:12:35.666 +00:00] [INFO] [datacoord/handler.go:299] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_1_447880825398107025v0] [posTs=449126111372640257] [posTime=2024/04/16 15:09:08.974 +00:00]
[2024/04/16 15:12:35.666 +00:00] [INFO] [datacoord/services.go:772] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=e3333646adb3dd4c4438d5c44e000bb1] [collectionID=447880825398107025] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_1_447880825398107025v0] ["# of unflushed segments"=0] ["# of flushed segments"=3] ["# of dropped segments"=1] ["# of indexed segments"=0]
[2024/04/16 15:12:35.665 +00:00] [INFO] [datacoord/services.go:757] ["get recovery info request received"] [traceID=74cfc868a3df9e7af83dfed1c8ac0418] [collectionID=444725728210909993] [partitionIDs="[]"]
[2024/04/16 15:12:35.666 +00:00] [INFO] [datacoord/handler.go:116] [GetQueryVChanPositions] [collectionID=444725728210909993] [channel=by-dev-rootcoord-dml_6_444725728210909993v0] [numOfSegments=6] ["indexed segment"=6]
[2024/04/16 15:12:35.666 +00:00] [INFO] [datacoord/handler.go:299] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_6_444725728210909993v0] [posTs=449126159240134657] [posTime=2024/04/16 15:12:11.574 +00:00]
[2024/04/16 15:12:35.666 +00:00] [INFO] [datacoord/services.go:772] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=74cfc868a3df9e7af83dfed1c8ac0418] [collectionID=444725728210909993] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_6_444725728210909993v0] ["# of unflushed segments"=0] ["# of flushed segments"=6] ["# of dropped segments"=0] ["# of indexed segments"=0]
[2024/04/16 15:12:35.665 +00:00] [INFO] [datacoord/handler.go:116] [GetQueryVChanPositions] [collectionID=447880825398106991] [channel=by-dev-rootcoord-dml_0_447880825398106991v0] [numOfSegments=2] ["indexed segment"=1]
[2024/04/16 15:12:35.666 +00:00] [INFO] [datacoord/handler.go:299] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_0_447880825398106991v0] [posTs=449126111372640257] [posTime=2024/04/16 15:09:08.974 +00:00]
[2024/04/16 15:12:35.666 +00:00] [INFO] [datacoord/services.go:772] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=a938811b88cdf6e58940ff02eb78ad3d] [collectionID=447880825398106991] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_0_447880825398106991v0] ["# of unflushed segments"=1] ["# of flushed segments"=1] ["# of dropped segments"=0] ["# of indexed segments"=0]
[2024/04/16 15:12:35.664 +00:00] [INFO] [datacoord/services.go:757] ["get recovery info request received"] [traceID=b3f2a02d01619a28f732933873048d7f] [collectionID=447042714314267305] [partitionIDs="[]"]
[2024/04/16 15:12:35.666 +00:00] [INFO] [datacoord/handler.go:116] [GetQueryVChanPositions] [collectionID=447042714314267305] [channel=by-dev-rootcoord-dml_7_447042714314267305v0] [numOfSegments=1] ["indexed segment"=1]
[2024/04/16 15:12:35.666 +00:00] [INFO] [datacoord/handler.go:299] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_7_447042714314267305v0] [posTs=449126153420537857] [posTime=2024/04/16 15:11:49.374 +00:00]
[2024/04/16 15:12:35.666 +00:00] [INFO] [datacoord/services.go:772] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=b3f2a02d01619a28f732933873048d7f] [collectionID=447042714314267305] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_7_447042714314267305v0] ["# of unflushed segments"=0] ["# of flushed segments"=1] ["# of dropped segments"=0] ["# of indexed segments"=0]
[2024/04/16 15:12:49.432 +00:00] [INFO] [proxy/meta_cache.go:900] ["expire all shard leader cache"] [database=default] [collections="[documents_EN,documents_DE,documents,document_texts,vectors_ml_e5_large,ai_helpdesk]"]
[2024/04/16 15:12:49.643 +00:00] [INFO] [proxy/meta_cache.go:900] ["expire all shard leader cache"] [database=default] [collections="[documents,document_texts,vectors_ml_e5_large,ai_helpdesk,documents_EN,documents_DE]"]
[2024/04/16 15:12:49.558 +00:00] [INFO] [observers/target_observer.go:450] ["Update readable segment version"] [collectionID=447880825398106991] [channelName=by-dev-rootcoord-dml_0_447880825398106991v0] [nodeID=12] [oldVersion=1713280345654142257] [newVersion=1713280355661739174]
[2024/04/16 15:12:49.411 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]

Anything else?

No response
@mihailyanchev mihailyanchev added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Apr 17, 2024
@yanliang567
Copy link
Contributor

@mihailyanchev the logs attached above seem everything is ok. According to your description, I'd suggest you allocate more resource for milvus container, as you have frequently delete and insert(upsert) operations to milvus.
Could you please attach the etcd backup for investigation？ Check this: https://github.com/milvus-io/birdwatcher for details about how to backup etcd with birdwatcher

/assign @mihailyanchev

@yanliang567 yanliang567 added triage/needs-information Indicates an issue needs more information in order to work on it. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Apr 17, 2024
@xiaofan-luan
Copy link
Contributor

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version: 2.3.4
- Deployment mode(standalone or cluster): standalone
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2): pymilvus 2.3.1
- OS(Ubuntu or CentOS): Ubuntu 22.04.2 LTS
- CPU/Memory: AWS t3.xlarge instance (4 cores, model: Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz and 16GB RAM, 100GB gp3 storage) 
- GPU: no GPU
- Others:

Current Behavior

We experienced two random shutdowns of the milvus-standalone container during periods of larger load to the Milvus (mainly insert and delete operations). There are no errors in the logs immediately before the shutdown. We observe 100% CPU utilisation right before the crash and the first time it happened the whole machine crashed too and restarted automatically. The minio and the etcd containers remain alive and running.

We have around 2-4 collections with no more than 200-300K records in each.

The first time this happened it was impossible to return the Milvus-standalone to normal. Every time we restart the container it would go back to 100% cpu utilisation and crash again. After several restarts it managed to remain stable, but it never managed to load its collections, so we had to rebuild everything from scratch.

I am wondering whether the resources are too limited for our setup and whether we have to either allocate more compute resources or move to the distributed mode.

Expected Behavior

I would expect some error log as to what is happening or some graceful way to let us know it needs more resources.

Steps To Reproduce

I have not been able to reproduce intentionally. It seems it is related to a longer period (several hours) of inserts and consecutive deletions of previously inserted records. 

Our pipeline requires to insert some records for immediate use, but then later on delete these records and replace them with new ones, once a second stage of processing has been completed.

Milvus Log

The log for this was 1.7GB so I am just posting the very begging and the very end before the crash.

The beginning:

2024/04/02 09:07:03 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined

    __  _________ _   ____  ______    
   /  |/  /  _/ /| | / / / / / __/    
  / /|_/ // // /_| |/ / /_/ /\ \    
 /_/  /_/___/____/___/\____/___/     

Welcome to use Milvus!
Version:   v2.3.4
Built:     Tue Jan  2 07:34:57 UTC 2024
GitCommit: 7a192da8
GoVersion: go version go1.20.7 linux/amd64

open pid file: /run/milvus/standalone.pid
lock pid file: /run/milvus/standalone.pid
[2024/04/02 09:07:03.415 +00:00] [INFO] [roles/roles.go:306] ["starting running Milvus components"]
[2024/04/02 09:07:03.415 +00:00] [INFO] [roles/roles.go:169] ["Enable Jemalloc"] ["Jemalloc Path"=/milvus/lib/libjemalloc.so]
[2024/04/02 09:07:03.434 +00:00] [DEBUG] [config/etcd_source.go:49] ["init etcd source"] [etcdInfo="{\"UseEmbed\":false,\"UseSSL\":false,\"Endpoints\":[\"etcd:2379\"],\"KeyPrefix\":\"by-dev\",\"CertFile\":\"/path/to/etcd-client.pem\",\"KeyFile\":\"/path/to/etcd-client-key.pem\",\"CaCertFile\":\"/path/to/ca.pem\",\"MinVersion\":\"1.3\",\"RefreshInterval\":5000000000}"]
[2024/04/02 09:07:03.435 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[etcd:2379]"] [minVersion=1.3]
[2024/04/02 09:07:03.435 +00:00] [INFO] [config/refresher.go:66] ["start refreshing configurations"] [source=FileSource]
[2024/04/02 09:07:03.437 +00:00] [DEBUG] [config/etcd_source.go:141] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[etcd:2379]"]
[2024/04/02 09:07:05.326 +00:00] [INFO] [config/refresher.go:66] ["start refreshing configurations"] [source=EtcdSource]
[2024/04/02 09:07:05.329 +00:00] [INFO] [config/refresher.go:66] ["start refreshing configurations"] [source=FileSource]
[2024/04/02 09:07:05.329 +00:00] [INFO] [paramtable/hook_config.go:21] ["hook config"] [hook={}]
[2024/04/02 09:07:05.329 +00:00] [INFO] [roles/roles.go:255] [setupPrometheusHTTPServer]
[2024/04/02 09:07:05.329 +00:00] [INFO] [http/server.go:112] ["management listen"] [addr=:9091]
[2024/04/02 09:07:05.330 +00:00] [INFO] [rootcoord/root_coord.go:157] ["update rootcoord state"] [state=Abnormal]
[2024/04/02 09:07:05.330 +00:00] [DEBUG] [rootcoord/service.go:166] ["init params done.."]
[2024/04/02 09:07:05.330 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[etcd:2379]"] [minVersion=1.3]
[2024/04/02 09:07:05.330 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[etcd:2379]"] [minVersion=1.3]
[2024/04/02 09:07:05.330 +00:00] [INFO] [components/index_coord.go:38] ["IndexCoord running ..."]
[2024/04/02 09:07:05.331 +00:00] [DEBUG] [querynode/service.go:100] [QueryNode] [port=21123]
[2024/04/02 09:07:05.331 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[etcd:2379]"] [minVersion=1.3]
[2024/04/02 09:07:05.331 +00:00] [DEBUG] [rootcoord/service.go:183] ["etcd connect done ..."]
[2024/04/02 09:07:05.331 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[etcd:2379]"] [minVersion=1.3]
[2024/04/02 09:07:05.331 +00:00] [DEBUG] [rootcoord/service.go:243] ["start grpc "] [port=53100]
[2024/04/02 09:07:05.332 +00:00] [DEBUG] [querynode/service.go:117] ["QueryNode connect to etcd successfully"]
[2024/04/02 09:07:05.332 +00:00] [DEBUG] [datacoord/service.go:147] ["network port"] [port=13333]
[2024/04/02 09:07:05.331 +00:00] [INFO] [datanode/service.go:245] ["DataNode address"] [address=172.21.0.5:21124]
[2024/04/02 09:07:05.332 +00:00] [DEBUG] [indexnode/indexnode.go:116] ["New IndexNode ..."]
[2024/04/02 09:07:05.332 +00:00] [DEBUG] [indexnode/service.go:87] [IndexNode] ["network address"=172.21.0.5:21121] ["network port: "=21121]
[2024/04/02 09:07:05.333 +00:00] [INFO] [proxy/lb_policy.go:78] ["use look_aside policy on replica selection"]
[2024/04/02 09:07:05.333 +00:00] [INFO] [runtime/asm_amd64.s:1598] ["Start check query node health loop"]
[2024/04/02 09:07:05.333 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[etcd:2379]"] [minVersion=1.3]
[2024/04/02 09:07:05.334 +00:00] [DEBUG] [querycoord/service.go:215] [network] [port=19531]

The end:

[2024/04/16 15:12:11.428 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
[2024/04/16 15:12:11.831 +00:00] [INFO] [datacoord/meta.go:1324] ["UpdateChannelCheckpoint done"] [vChannel=by-dev-rootcoord-dml_3_444725728207049774v0] [ts=449126159240134657] [msgID="\ufffd\ufffdsjgx:\u0006"] [time=2024/04/16 15:12:11.574 +00:00]
[2024/04/16 15:12:11.834 +00:00] [INFO] [datacoord/meta.go:1324] ["UpdateChannelCheckpoint done"] [vChannel=by-dev-rootcoord-dml_0_447880825398106991v0] [ts=449126111372640257] [msgID=")\ufffdsjgx:\u0006"] [time=2024/04/16 15:09:08.974 +00:00]
[2024/04/16 15:12:11.835 +00:00] [INFO] [datacoord/meta.go:1324] ["UpdateChannelCheckpoint done"] [vChannel=by-dev-rootcoord-dml_2_444725728207049736v0] [ts=449126159240134657] [msgID="\ufffd\ufffdsjgx:\u0006"] [time=2024/04/16 15:12:11.574 +00:00]
[2024/04/16 15:12:11.837 +00:00] [INFO] [datacoord/meta.go:1324] ["UpdateChannelCheckpoint done"] [vChannel=by-dev-rootcoord-dml_6_444725728210909993v0] [ts=449126159240134657] [msgID="\ufffd\ufffdsjgx:\u0006"] [time=2024/04/16 15:12:11.574 +00:00]
[2024/04/16 15:12:12.429 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
[2024/04/16 15:12:12.911 +00:00] [INFO] [proxy/meta_cache.go:900] ["expire all shard leader cache"] [database=default] [collections="[vectors_ml_e5_large,ai_helpdesk,documents_EN,documents_DE,documents,document_texts]"]
[2024/04/16 15:12:13.428 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
[2024/04/16 15:12:14.434 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
[2024/04/16 15:12:15.429 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
[2024/04/16 15:12:15.649 +00:00] [INFO] [datacoord/index_service.go:513] ["receive DescribeIndex request"] [traceID=f19bbbd5696a39f07e531de444475694] [collectionID=447880825398107025] [indexName=] [timestamp=0]
[2024/04/16 15:12:15.650 +00:00] [INFO] [datacoord/index_service.go:513] ["receive DescribeIndex request"] [traceID=8838bd562f709815d9aa5a4865043ca6] [collectionID=444725728210909993] [indexName=] [timestamp=0]
[2024/04/16 15:12:15.650 +00:00] [INFO] [datacoord/index_service.go:449] ["completeIndexInfo success"] [collectionID=444725728210909993] [indexID=447042714307813983] [totalRows=530271] [indexRows=530271] [pendingIndexRows=0] [state=Finished] [failReason=]
[2024/04/16 15:12:15.651 +00:00] [INFO] [datacoord/index_service.go:560] ["DescribeIndex success"] [traceID=8838bd562f709815d9aa5a4865043ca6] [collectionID=444725728210909993] [indexName=]
[2024/04/16 15:12:15.650 +00:00] [INFO] [datacoord/index_service.go:449] ["completeIndexInfo success"] [collectionID=447880825398107025] [indexID=447880825398107030] [totalRows=272665] [indexRows=272665] [pendingIndexRows=129433] [state=Finished] [failReason=]
[2024/04/16 15:12:15.651 +00:00] [INFO] [datacoord/index_service.go:560] ["DescribeIndex success"] [traceID=f19bbbd5696a39f07e531de444475694] [collectionID=447880825398107025] [indexName=]
[2024/04/16 15:12:15.650 +00:00] [INFO] [datacoord/index_service.go:513] ["receive DescribeIndex request"] [traceID=5c5ef398d78f7715ad1d82b5626d3bdc] [collectionID=444725728207049736] [indexName=] [timestamp=0]
[2024/04/16 15:12:15.652 +00:00] [INFO] [datacoord/index_service.go:449] ["completeIndexInfo success"] [collectionID=444725728207049736] [indexID=444725728207049740] [totalRows=9069] [indexRows=9069] [pendingIndexRows=0] [state=Finished] [failReason=]
[2024/04/16 15:12:15.652 +00:00] [INFO] [datacoord/index_service.go:560] ["DescribeIndex success"] [traceID=5c5ef398d78f7715ad1d82b5626d3bdc] [collectionID=444725728207049736] [indexName=]
[2024/04/16 15:12:15.654 +00:00] [INFO] [querynodev2/services.go:1366] ["sync action"] [traceID=506701dfa13daae0cf8e988850b7e0f6] [collectionID=444725728210909993] [channel=by-dev-rootcoord-dml_6_444725728210909993v0] [currentNodeID=12] [Action=UpdateVersion] [TargetVersion=1713280325656191319]
[2024/04/16 15:12:15.650 +00:00] [INFO] [datacoord/index_service.go:513] ["receive DescribeIndex request"] [traceID=646294324f36ec35e92a16d8ad6f6190] [collectionID=447042714314267305] [indexName=] [timestamp=0]
[2024/04/16 15:12:15.650 +00:00] [INFO] [datacoord/index_service.go:513] ["receive DescribeIndex request"] [traceID=ffc76ddf1363520ca0595062c1696199] [collectionID=447880825398106991] [indexName=] [timestamp=0]
[2024/04/16 15:12:15.650 +00:00] [INFO] [datacoord/index_service.go:513] ["receive DescribeIndex request"] [traceID=63ec49683d8af69062e072cb2e9880b5] [collectionID=444725728207049774] [indexName=] [timestamp=0]
[2024/04/16 15:12:15.655 +00:00] [INFO] [querynodev2/services.go:1366] ["sync action"] [traceID=5bfb7260f6cae5528f1f2e3ef0e329df] [collectionID=444725728207049736] [channel=by-dev-rootcoord-dml_2_444725728207049736v0] [currentNodeID=12] [Action=UpdateVersion] [TargetVersion=1713280325658861540]
[2024/04/16 15:12:15.655 +00:00] [INFO] [querynodev2/services.go:1366] ["sync action"] [traceID=3f6befb4928d5e8e44932a3ef7ce9ae5] [collectionID=447880825398107025] [channel=by-dev-rootcoord-dml_1_447880825398107025v0] [currentNodeID=12] [Action=UpdateVersion] [TargetVersion=1713280325654978717]
[2024/04/16 15:12:15.657 +00:00] [INFO] [delegator/distribution.go:294] ["Update readable segment version"] [oldVersion=1713280315653425196] [newVersion=1713280325654978717] [growingSegmentNum=0] [sealedSegmentNum=3]
[2024/04/16 15:12:15.657 +00:00] [INFO] [delegator/distribution.go:294] ["Update readable segment version"] [oldVersion=1713280315662337894] [newVersion=1713280325656191319] [growingSegmentNum=0] [sealedSegmentNum=6]
[2024/04/16 15:12:15.658 +00:00] [INFO] [datacoord/index_service.go:449] ["completeIndexInfo success"] [collectionID=444725728207049774] [indexID=444725728207049778] [totalRows=9069] [indexRows=9069] [pendingIndexRows=0] [state=Finished] [failReason=]
[2024/04/16 15:12:15.658 +00:00] [INFO] [datacoord/index_service.go:560] ["DescribeIndex success"] [traceID=63ec49683d8af69062e072cb2e9880b5] [collectionID=444725728207049774] [indexName=]
[2024/04/16 15:12:15.658 +00:00] [INFO] [datacoord/index_service.go:449] ["completeIndexInfo success"] [collectionID=447042714314267305] [indexID=447042714314267311] [totalRows=35] [indexRows=35] [pendingIndexRows=0] [state=Finished] [failReason=]
[2024/04/16 15:12:15.658 +00:00] [INFO] [datacoord/index_service.go:560] ["DescribeIndex success"] [traceID=646294324f36ec35e92a16d8ad6f6190] [collectionID=447042714314267305] [indexName=]
[2024/04/16 15:12:15.658 +00:00] [INFO] [datacoord/index_service.go:449] ["completeIndexInfo success"] [collectionID=447880825398106991] [indexID=447880825398106997] [totalRows=271127] [indexRows=271127] [pendingIndexRows=0] [state=Finished] [failReason=]
[2024/04/16 15:12:15.659 +00:00] [INFO] [datacoord/index_service.go:560] ["DescribeIndex success"] [traceID=ffc76ddf1363520ca0595062c1696199] [collectionID=447880825398106991] [indexName=]
[2024/04/16 15:12:15.658 +00:00] [INFO] [delegator/distribution.go:294] ["Update readable segment version"] [oldVersion=1713280315664583265] [newVersion=1713280325658861540] [growingSegmentNum=0] [sealedSegmentNum=2]
[2024/04/16 15:12:15.659 +00:00] [INFO] [observers/target_observer.go:473] ["observer trigger update current target"] [collectionID=444725728207049736]
[2024/04/16 15:12:15.660 +00:00] [INFO] [datacoord/services.go:757] ["get recovery info request received"] [traceID=eb1fa7d81d3e30624dd0260d48dda8fe] [collectionID=447880825398107025] [partitionIDs="[]"]
[2024/04/16 15:12:15.662 +00:00] [INFO] [datacoord/handler.go:116] [GetQueryVChanPositions] [collectionID=447880825398107025] [channel=by-dev-rootcoord-dml_1_447880825398107025v0] [numOfSegments=6] ["indexed segment"=4]
[2024/04/16 15:12:15.661 +00:00] [INFO] [querynodev2/services.go:1366] ["sync action"] [traceID=eec5b9edd984f116febea1cc2dedda55] [collectionID=447042714314267305] [channel=by-dev-rootcoord-dml_7_447042714314267305v0] [currentNodeID=12] [Action=UpdateVersion] [TargetVersion=1713280325659637243]
[2024/04/16 15:12:15.661 +00:00] [INFO] [datacoord/services.go:757] ["get recovery info request received"] [traceID=ca598e81ec98311b57263b6018fdf23b] [collectionID=444725728210909993] [partitionIDs="[]"]
[2024/04/16 15:12:15.662 +00:00] [INFO] [querynodev2/services.go:1366] ["sync action"] [traceID=3fd75d940755f2f29b69186c88913f27] [collectionID=444725728207049774] [channel=by-dev-rootcoord-dml_3_444725728207049774v0] [currentNodeID=12] [Action=UpdateVersion] [TargetVersion=1713280325657230648]
[2024/04/16 15:12:15.663 +00:00] [INFO] [delegator/distribution.go:294] ["Update readable segment version"] [oldVersion=1713280315653937742] [newVersion=1713280325657230648] [growingSegmentNum=0] [sealedSegmentNum=2]
[2024/04/16 15:12:15.663 +00:00] [INFO] [datacoord/handler.go:116] [GetQueryVChanPositions] [collectionID=444725728210909993] [channel=by-dev-rootcoord-dml_6_444725728210909993v0] [numOfSegments=6] ["indexed segment"=6]
[2024/04/16 15:12:15.663 +00:00] [INFO] [datacoord/handler.go:299] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_6_444725728210909993v0] [posTs=449126159240134657] [posTime=2024/04/16 15:12:11.574 +00:00]
[2024/04/16 15:12:15.663 +00:00] [INFO] [datacoord/services.go:772] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=ca598e81ec98311b57263b6018fdf23b] [collectionID=444725728210909993] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_6_444725728210909993v0] ["# of unflushed segments"=0] ["# of flushed segments"=6] ["# of dropped segments"=0] ["# of indexed segments"=0]
[2024/04/16 15:12:15.663 +00:00] [INFO] [datacoord/handler.go:299] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_1_447880825398107025v0] [posTs=449126111372640257] [posTime=2024/04/16 15:09:08.974 +00:00]
[2024/04/16 15:12:15.664 +00:00] [INFO] [datacoord/services.go:772] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=eb1fa7d81d3e30624dd0260d48dda8fe] [collectionID=447880825398107025] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_1_447880825398107025v0] ["# of unflushed segments"=0] ["# of flushed segments"=3] ["# of dropped segments"=1] ["# of indexed segments"=0]
[2024/04/16 15:12:15.662 +00:00] [INFO] [datacoord/services.go:757] ["get recovery info request received"] [traceID=3d36a58f4deb6f100332872f8e8a7a15] [collectionID=444725728207049736] [partitionIDs="[]"]
[2024/04/16 15:12:15.663 +00:00] [INFO] [delegator/distribution.go:294] ["Update readable segment version"] [oldVersion=1713280315661582892] [newVersion=1713280325659637243] [growingSegmentNum=0] [sealedSegmentNum=1]
[2024/04/16 15:12:15.664 +00:00] [INFO] [querynodev2/services.go:1366] ["sync action"] [traceID=e0fda5bf84419d01ec0b1de09e4eca32] [collectionID=447880825398106991] [channel=by-dev-rootcoord-dml_0_447880825398106991v0] [currentNodeID=12] [Action=UpdateVersion] [TargetVersion=1713280325655641871]
[2024/04/16 15:12:15.665 +00:00] [INFO] [delegator/distribution.go:294] ["Update readable segment version"] [oldVersion=1713280315653371467] [newVersion=1713280325655641871] [growingSegmentNum=1] [sealedSegmentNum=1]
[2024/04/16 15:12:15.665 +00:00] [INFO] [datacoord/handler.go:116] [GetQueryVChanPositions] [collectionID=444725728207049736] [channel=by-dev-rootcoord-dml_2_444725728207049736v0] [numOfSegments=2] ["indexed segment"=2]
[2024/04/16 15:12:15.667 +00:00] [INFO] [datacoord/handler.go:299] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_2_444725728207049736v0] [posTs=449126159240134657] [posTime=2024/04/16 15:12:11.574 +00:00]
[2024/04/16 15:12:15.667 +00:00] [INFO] [datacoord/services.go:772] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=3d36a58f4deb6f100332872f8e8a7a15] [collectionID=444725728207049736] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_2_444725728207049736v0] ["# of unflushed segments"=0] ["# of flushed segments"=2] ["# of dropped segments"=0] ["# of indexed segments"=0]
[2024/04/16 15:12:15.666 +00:00] [INFO] [datacoord/services.go:757] ["get recovery info request received"] [traceID=210583f2f932c873de2badbc03bdef73] [collectionID=444725728207049774] [partitionIDs="[]"]
[2024/04/16 15:12:15.667 +00:00] [INFO] [datacoord/handler.go:116] [GetQueryVChanPositions] [collectionID=444725728207049774] [channel=by-dev-rootcoord-dml_3_444725728207049774v0] [numOfSegments=2] ["indexed segment"=2]
[2024/04/16 15:12:15.666 +00:00] [INFO] [datacoord/services.go:757] ["get recovery info request received"] [traceID=679067e6d3640f3166fe4f0bded06a51] [collectionID=447880825398106991] [partitionIDs="[]"]
[2024/04/16 15:12:15.666 +00:00] [INFO] [datacoord/services.go:757] ["get recovery info request received"] [traceID=9b33e06be60de7980727a7c56eee4606] [collectionID=447042714314267305] [partitionIDs="[]"]
[2024/04/16 15:12:15.668 +00:00] [INFO] [datacoord/handler.go:299] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_3_444725728207049774v0] [posTs=449126159240134657] [posTime=2024/04/16 15:12:11.574 +00:00]
[2024/04/16 15:12:15.668 +00:00] [INFO] [datacoord/services.go:772] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=210583f2f932c873de2badbc03bdef73] [collectionID=444725728207049774] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_3_444725728207049774v0] ["# of unflushed segments"=0] ["# of flushed segments"=2] ["# of dropped segments"=0] ["# of indexed segments"=0]
[2024/04/16 15:12:15.668 +00:00] [INFO] [datacoord/handler.go:116] [GetQueryVChanPositions] [collectionID=447880825398106991] [channel=by-dev-rootcoord-dml_0_447880825398106991v0] [numOfSegments=2] ["indexed segment"=1]
[2024/04/16 15:12:15.669 +00:00] [INFO] [datacoord/handler.go:299] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_0_447880825398106991v0] [posTs=449126111372640257] [posTime=2024/04/16 15:09:08.974 +00:00]
[2024/04/16 15:12:15.669 +00:00] [INFO] [datacoord/services.go:772] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=679067e6d3640f3166fe4f0bded06a51] [collectionID=447880825398106991] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_0_447880825398106991v0] ["# of unflushed segments"=1] ["# of flushed segments"=1] ["# of dropped segments"=0] ["# of indexed segments"=0]
[2024/04/16 15:12:15.668 +00:00] [INFO] [datacoord/handler.go:116] [GetQueryVChanPositions] [collectionID=447042714314267305] [channel=by-dev-rootcoord-dml_7_447042714314267305v0] [numOfSegments=1] ["indexed segment"=1]
[2024/04/16 15:12:15.669 +00:00] [INFO] [datacoord/handler.go:299] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_7_447042714314267305v0] [posTs=449126153420537857] [posTime=2024/04/16 15:11:49.374 +00:00]
[2024/04/16 15:12:15.669 +00:00] [INFO] [datacoord/services.go:772] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=9b33e06be60de7980727a7c56eee4606] [collectionID=447042714314267305] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_7_447042714314267305v0] ["# of unflushed segments"=0] ["# of flushed segments"=1] ["# of dropped segments"=0] ["# of indexed segments"=0]
[2024/04/16 15:12:15.912 +00:00] [INFO] [proxy/meta_cache.go:900] ["expire all shard leader cache"] [database=default] [collections="[documents_DE,documents,document_texts,vectors_ml_e5_large,ai_helpdesk,documents_EN]"]
[2024/04/16 15:12:16.428 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
[2024/04/16 15:12:17.428 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
[2024/04/16 15:12:18.428 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
[2024/04/16 15:12:18.910 +00:00] [INFO] [proxy/meta_cache.go:900] ["expire all shard leader cache"] [database=default] [collections="[documents,document_texts,vectors_ml_e5_large,ai_helpdesk,documents_EN,documents_DE]"]
[2024/04/16 15:12:19.433 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
[2024/04/16 15:12:20.431 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
[2024/04/16 15:12:21.429 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
[2024/04/16 15:12:21.911 +00:00] [INFO] [proxy/meta_cache.go:900] ["expire all shard leader cache"] [database=default] [collections="[document_texts,vectors_ml_e5_large,ai_helpdesk,documents_EN,documents_DE,documents]"]
[2024/04/16 15:12:22.428 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
[2024/04/16 15:12:23.432 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
[2024/04/16 15:12:24.429 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
[2024/04/16 15:12:24.913 +00:00] [INFO] [proxy/meta_cache.go:900] ["expire all shard leader cache"] [database=default] [collections="[documents_EN,documents_DE,documents,document_texts,vectors_ml_e5_large,ai_helpdesk]"]
[2024/04/16 15:12:25.428 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
[2024/04/16 15:12:25.649 +00:00] [INFO] [datacoord/index_service.go:513] ["receive DescribeIndex request"] [traceID=f64fc74364475dd268c64da4e4465858] [collectionID=447042714314267305] [indexName=] [timestamp=0]
[2024/04/16 15:12:25.649 +00:00] [INFO] [datacoord/index_service.go:449] ["completeIndexInfo success"] [collectionID=447042714314267305] [indexID=447042714314267311] [totalRows=35] [indexRows=35] [pendingIndexRows=0] [state=Finished] [failReason=]
[2024/04/16 15:12:25.649 +00:00] [INFO] [datacoord/index_service.go:560] ["DescribeIndex success"] [traceID=f64fc74364475dd268c64da4e4465858] [collectionID=447042714314267305] [indexName=]
[2024/04/16 15:12:25.649 +00:00] [INFO] [datacoord/index_service.go:513] ["receive DescribeIndex request"] [traceID=f6f435c6cd6cde94b6d94dd7af0aa1e2] [collectionID=444725728207049736] [indexName=] [timestamp=0]
[2024/04/16 15:12:25.649 +00:00] [INFO] [datacoord/index_service.go:449] ["completeIndexInfo success"] [collectionID=444725728207049736] [indexID=444725728207049740] [totalRows=9069] [indexRows=9069] [pendingIndexRows=0] [state=Finished] [failReason=]
[2024/04/16 15:12:25.649 +00:00] [INFO] [datacoord/index_service.go:560] ["DescribeIndex success"] [traceID=f6f435c6cd6cde94b6d94dd7af0aa1e2] [collectionID=444725728207049736] [indexName=]
[2024/04/16 15:12:25.649 +00:00] [INFO] [datacoord/index_service.go:513] ["receive DescribeIndex request"] [traceID=03cc858ae36eda8b4a02c1563029d716] [collectionID=444725728210909993] [indexName=] [timestamp=0]
[2024/04/16 15:12:25.649 +00:00] [INFO] [datacoord/index_service.go:449] ["completeIndexInfo success"] [collectionID=444725728210909993] [indexID=447042714307813983] [totalRows=530271] [indexRows=530271] [pendingIndexRows=0] [state=Finished] [failReason=]
[2024/04/16 15:12:25.649 +00:00] [INFO] [datacoord/index_service.go:560] ["DescribeIndex success"] [traceID=03cc858ae36eda8b4a02c1563029d716] [collectionID=444725728210909993] [indexName=]
[2024/04/16 15:12:25.650 +00:00] [INFO] [datacoord/index_service.go:513] ["receive DescribeIndex request"] [traceID=11f3315add47168bb6face6c9b059aea] [collectionID=444725728207049774] [indexName=] [timestamp=0]
[2024/04/16 15:12:25.650 +00:00] [INFO] [datacoord/index_service.go:449] ["completeIndexInfo success"] [collectionID=444725728207049774] [indexID=444725728207049778] [totalRows=9069] [indexRows=9069] [pendingIndexRows=0] [state=Finished] [failReason=]
[2024/04/16 15:12:25.650 +00:00] [INFO] [datacoord/index_service.go:560] ["DescribeIndex success"] [traceID=11f3315add47168bb6face6c9b059aea] [collectionID=444725728207049774] [indexName=]
[2024/04/16 15:12:25.650 +00:00] [INFO] [querynodev2/services.go:1366] ["sync action"] [traceID=1d5c6e96807061f1e249897bf0dbb9fb] [collectionID=447042714314267305] [channel=by-dev-rootcoord-dml_7_447042714314267305v0] [currentNodeID=12] [Action=UpdateVersion] [TargetVersion=1713280335666048447]
[2024/04/16 15:12:25.650 +00:00] [INFO] [delegator/distribution.go:294] ["Update readable segment version"] [oldVersion=1713280325659637243] [newVersion=1713280335666048447] [growingSegmentNum=0] [sealedSegmentNum=1]
[2024/04/16 15:12:25.650 +00:00] [INFO] [datacoord/index_service.go:513] ["receive DescribeIndex request"] [traceID=79ea0048efa7b86ccaea8cf6bf7e6a40] [collectionID=447880825398106991] [indexName=] [timestamp=0]
[2024/04/16 15:12:25.651 +00:00] [INFO] [datacoord/index_service.go:449] ["completeIndexInfo success"] [collectionID=447880825398106991] [indexID=447880825398106997] [totalRows=271127] [indexRows=271127] [pendingIndexRows=0] [state=Finished] [failReason=]
[2024/04/16 15:12:25.651 +00:00] [INFO] [datacoord/index_service.go:560] ["DescribeIndex success"] [traceID=79ea0048efa7b86ccaea8cf6bf7e6a40] [collectionID=447880825398106991] [indexName=]
[2024/04/16 15:12:25.650 +00:00] [INFO] [datacoord/index_service.go:513] ["receive DescribeIndex request"] [traceID=6da94005a6f650a753c4380f20d055e3] [collectionID=447880825398107025] [indexName=] [timestamp=0]
[2024/04/16 15:12:25.651 +00:00] [INFO] [datacoord/index_service.go:449] ["completeIndexInfo success"] [collectionID=447880825398107025] [indexID=447880825398107030] [totalRows=272665] [indexRows=272665] [pendingIndexRows=129433] [state=Finished] [failReason=]
[2024/04/16 15:12:25.651 +00:00] [INFO] [datacoord/index_service.go:560] ["DescribeIndex success"] [traceID=6da94005a6f650a753c4380f20d055e3] [collectionID=447880825398107025] [indexName=]
[2024/04/16 15:12:25.651 +00:00] [INFO] [querynodev2/services.go:1366] ["sync action"] [traceID=94f9eb439f1c52a27eef0b77be8da23a] [collectionID=444725728207049736] [channel=by-dev-rootcoord-dml_2_444725728207049736v0] [currentNodeID=12] [Action=UpdateVersion] [TargetVersion=1713280335661490246]
[2024/04/16 15:12:25.651 +00:00] [INFO] [delegator/distribution.go:294] ["Update readable segment version"] [oldVersion=1713280325658861540] [newVersion=1713280335661490246] [growingSegmentNum=0] [sealedSegmentNum=2]
[2024/04/16 15:12:25.651 +00:00] [INFO] [querynodev2/services.go:1366] ["sync action"] [traceID=c0adadb3ce6c1ef0fcf5658486aa9141] [collectionID=444725728207049774] [channel=by-dev-rootcoord-dml_3_444725728207049774v0] [currentNodeID=12] [Action=UpdateVersion] [TargetVersion=1713280335664399979]
[2024/04/16 15:12:25.652 +00:00] [INFO] [delegator/distribution.go:294] ["Update readable segment version"] [oldVersion=1713280325657230648] [newVersion=1713280335664399979] [growingSegmentNum=0] [sealedSegmentNum=2]
[2024/04/16 15:12:25.652 +00:00] [INFO] [querynodev2/services.go:1366] ["sync action"] [traceID=3dc7c6d4c35f19746db24081e4c3a7ef] [collectionID=444725728210909993] [channel=by-dev-rootcoord-dml_6_444725728210909993v0] [currentNodeID=12] [Action=UpdateVersion] [TargetVersion=1713280335661133232]
[2024/04/16 15:12:25.653 +00:00] [INFO] [delegator/distribution.go:294] ["Update readable segment version"] [oldVersion=1713280325656191319] [newVersion=1713280335661133232] [growingSegmentNum=0] [sealedSegmentNum=6]
[2024/04/16 15:12:25.653 +00:00] [INFO] [querynodev2/services.go:1366] ["sync action"] [traceID=a4356cfe07ba0d0c89131c02db36e757] [collectionID=447880825398106991] [channel=by-dev-rootcoord-dml_0_447880825398106991v0] [currentNodeID=12] [Action=UpdateVersion] [TargetVersion=1713280335666636444]
[2024/04/16 15:12:25.653 +00:00] [INFO] [delegator/distribution.go:294] ["Update readable segment version"] [oldVersion=1713280325655641871] [newVersion=1713280335666636444] [growingSegmentNum=1] [sealedSegmentNum=1]
[2024/04/16 15:12:25.653 +00:00] [INFO] [observers/target_observer.go:473] ["observer trigger update current target"] [collectionID=447042714314267305]
[2024/04/16 15:12:25.653 +00:00] [INFO] [querynodev2/services.go:1366] ["sync action"] [traceID=d81fc6efd74998763e728f64ed1fed6e] [collectionID=447880825398107025] [channel=by-dev-rootcoord-dml_1_447880825398107025v0] [currentNodeID=12] [Action=UpdateVersion] [TargetVersion=1713280335660187976]
[2024/04/16 15:12:25.654 +00:00] [INFO] [delegator/distribution.go:294] ["Update readable segment version"] [oldVersion=1713280325654978717] [newVersion=1713280335660187976] [growingSegmentNum=0] [sealedSegmentNum=3]
[2024/04/16 15:12:25.653 +00:00] [INFO] [datacoord/services.go:757] ["get recovery info request received"] [traceID=7ed73cc39f75d9f12e3de83c340f420c] [collectionID=447042714314267305] [partitionIDs="[]"]
[2024/04/16 15:12:25.654 +00:00] [INFO] [datacoord/handler.go:116] [GetQueryVChanPositions] [collectionID=447042714314267305] [channel=by-dev-rootcoord-dml_7_447042714314267305v0] [numOfSegments=1] ["indexed segment"=1]
[2024/04/16 15:12:25.654 +00:00] [INFO] [datacoord/handler.go:299] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_7_447042714314267305v0] [posTs=449126153420537857] [posTime=2024/04/16 15:11:49.374 +00:00]
[2024/04/16 15:12:25.655 +00:00] [INFO] [datacoord/services.go:772] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=7ed73cc39f75d9f12e3de83c340f420c] [collectionID=447042714314267305] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_7_447042714314267305v0] ["# of unflushed segments"=0] ["# of flushed segments"=1] ["# of dropped segments"=0] ["# of indexed segments"=0]
[2024/04/16 15:12:25.654 +00:00] [INFO] [datacoord/services.go:757] ["get recovery info request received"] [traceID=791315521ffdd06f0ad1a21d07fe3cdf] [collectionID=444725728210909993] [partitionIDs="[]"]
[2024/04/16 15:12:25.656 +00:00] [INFO] [datacoord/handler.go:116] [GetQueryVChanPositions] [collectionID=444725728210909993] [channel=by-dev-rootcoord-dml_6_444725728210909993v0] [numOfSegments=6] ["indexed segment"=6]
[2024/04/16 15:12:25.656 +00:00] [INFO] [datacoord/handler.go:299] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_6_444725728210909993v0] [posTs=449126159240134657] [posTime=2024/04/16 15:12:11.574 +00:00]
[2024/04/16 15:12:25.656 +00:00] [INFO] [datacoord/services.go:772] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=791315521ffdd06f0ad1a21d07fe3cdf] [collectionID=444725728210909993] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_6_444725728210909993v0] ["# of unflushed segments"=0] ["# of flushed segments"=6] ["# of dropped segments"=0] ["# of indexed segments"=0]
[2024/04/16 15:12:25.655 +00:00] [INFO] [datacoord/services.go:757] ["get recovery info request received"] [traceID=d44e88f220f5e73cbb375a9e15af364c] [collectionID=447880825398106991] [partitionIDs="[]"]
[2024/04/16 15:12:25.657 +00:00] [INFO] [datacoord/handler.go:116] [GetQueryVChanPositions] [collectionID=447880825398106991] [channel=by-dev-rootcoord-dml_0_447880825398106991v0] [numOfSegments=2] ["indexed segment"=1]
[2024/04/16 15:12:25.657 +00:00] [INFO] [datacoord/handler.go:299] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_0_447880825398106991v0] [posTs=449126111372640257] [posTime=2024/04/16 15:09:08.974 +00:00]
[2024/04/16 15:12:25.657 +00:00] [INFO] [datacoord/services.go:772] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=d44e88f220f5e73cbb375a9e15af364c] [collectionID=447880825398106991] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_0_447880825398106991v0] ["# of unflushed segments"=1] ["# of flushed segments"=1] ["# of dropped segments"=0] ["# of indexed segments"=0]
[2024/04/16 15:12:25.655 +00:00] [INFO] [datacoord/services.go:757] ["get recovery info request received"] [traceID=3f8eb5c2e9571f887956e83162541ebe] [collectionID=444725728207049736] [partitionIDs="[]"]
[2024/04/16 15:12:25.658 +00:00] [INFO] [datacoord/handler.go:116] [GetQueryVChanPositions] [collectionID=444725728207049736] [channel=by-dev-rootcoord-dml_2_444725728207049736v0] [numOfSegments=2] ["indexed segment"=2]
[2024/04/16 15:12:25.658 +00:00] [INFO] [datacoord/handler.go:299] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_2_444725728207049736v0] [posTs=449126159240134657] [posTime=2024/04/16 15:12:11.574 +00:00]
[2024/04/16 15:12:25.655 +00:00] [INFO] [datacoord/services.go:757] ["get recovery info request received"] [traceID=68f4bc48d2c9d4a703d313b5b7535c7f] [collectionID=444725728207049774] [partitionIDs="[]"]
[2024/04/16 15:12:25.655 +00:00] [INFO] [datacoord/services.go:757] ["get recovery info request received"] [traceID=34f21350a04ed9602e8470e4d524fe8f] [collectionID=447880825398107025] [partitionIDs="[]"]
[2024/04/16 15:12:25.658 +00:00] [INFO] [datacoord/handler.go:116] [GetQueryVChanPositions] [collectionID=447880825398107025] [channel=by-dev-rootcoord-dml_1_447880825398107025v0] [numOfSegments=6] ["indexed segment"=4]
[2024/04/16 15:12:25.659 +00:00] [INFO] [datacoord/handler.go:299] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_1_447880825398107025v0] [posTs=449126111372640257] [posTime=2024/04/16 15:09:08.974 +00:00]
[2024/04/16 15:12:25.659 +00:00] [INFO] [datacoord/services.go:772] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=34f21350a04ed9602e8470e4d524fe8f] [collectionID=447880825398107025] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_1_447880825398107025v0] ["# of unflushed segments"=0] ["# of flushed segments"=3] ["# of dropped segments"=1] ["# of indexed segments"=0]
[2024/04/16 15:12:25.659 +00:00] [INFO] [datacoord/services.go:772] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=3f8eb5c2e9571f887956e83162541ebe] [collectionID=444725728207049736] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_2_444725728207049736v0] ["# of unflushed segments"=0] ["# of flushed segments"=2] ["# of dropped segments"=0] ["# of indexed segments"=0]
[2024/04/16 15:12:25.659 +00:00] [INFO] [datacoord/handler.go:116] [GetQueryVChanPositions] [collectionID=444725728207049774] [channel=by-dev-rootcoord-dml_3_444725728207049774v0] [numOfSegments=2] ["indexed segment"=2]
[2024/04/16 15:12:25.660 +00:00] [INFO] [datacoord/handler.go:299] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_3_444725728207049774v0] [posTs=449126159240134657] [posTime=2024/04/16 15:12:11.574 +00:00]
[2024/04/16 15:12:25.660 +00:00] [INFO] [datacoord/services.go:772] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=68f4bc48d2c9d4a703d313b5b7535c7f] [collectionID=444725728207049774] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_3_444725728207049774v0] ["# of unflushed segments"=0] ["# of flushed segments"=2] ["# of dropped segments"=0] ["# of indexed segments"=0]
[2024/04/16 15:12:26.429 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
[2024/04/16 15:12:27.429 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
[2024/04/16 15:12:27.697 +00:00] [INFO] [proxy/meta_cache.go:802] ["no shard cache for collection, try to get shard leaders from QueryCoord"] [traceID=7cf7a80fcc0ab91d8032a48f90ee07fa] [collectionName=document_texts] [collectionID=447880825398106991]
[2024/04/16 15:12:27.705 +00:00] [INFO] [datacoord/services.go:174] ["handle assign segment request"] [traceID=caecc67a21e3fab2e6fa9112cc4a2bc2] [collectionID=447880825398106991] [partitionID=447880825398106992] [channelName=by-dev-rootcoord-dml_0_447880825398106991v0] [count=7] [isImport=false] ["import task ID"=0]
[2024/04/16 15:12:27.706 +00:00] [INFO] [datacoord/meta.go:938] ["meta update: add allocation - complete"] [segmentID=448803498411568777]
[2024/04/16 15:12:27.706 +00:00] [INFO] [datacoord/services.go:213] ["success to assign segments"] [traceID=caecc67a21e3fab2e6fa9112cc4a2bc2] [collectionID=447880825398106991] [assignments="[{\"SegmentID\":448803498411568777,\"NumOfRows\":7,\"ExpireTime\":449126163984941059}]"]
[2024/04/16 15:12:27.712 +00:00] [INFO] [datacoord/services.go:174] ["handle assign segment request"] [traceID=afd43dc810910dcf4a01f398b656ee7d] [collectionID=447880825398107025] [partitionID=447880825398107026] [channelName=by-dev-rootcoord-dml_1_447880825398107025v0] [count=7] [isImport=false] ["import task ID"=0]
[2024/04/16 15:12:27.713 +00:00] [INFO] [datacoord/meta.go:938] ["meta update: add allocation - complete"] [segmentID=448803498411599794]
[2024/04/16 15:12:27.713 +00:00] [INFO] [datacoord/services.go:213] ["success to assign segments"] [traceID=afd43dc810910dcf4a01f398b656ee7d] [collectionID=447880825398107025] [assignments="[{\"SegmentID\":448803498411599794,\"NumOfRows\":7,\"ExpireTime\":449126163984941061}]"]
[2024/04/16 15:12:27.911 +00:00] [INFO] [proxy/meta_cache.go:900] ["expire all shard leader cache"] [database=default] [collections="[documents,document_texts,vectors_ml_e5_large,ai_helpdesk,documents_EN,documents_DE]"]
[2024/04/16 15:12:28.026 +00:00] [INFO] [datanode/channel_meta.go:608] ["updated segment num row"] [segmentID=448803498411599794] [addedNumRows=7] [numRowsSum=58]
[2024/04/16 15:12:28.030 +00:00] [INFO] [datanode/channel_meta.go:634] ["updated segment memorySize"] [segmentID=448803498411599794] [memorySize=249230]
[2024/04/16 15:12:28.030 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:286] ["segment buffer status"] [no.=0] [segmentID=448803498411599794] [channel=by-dev-rootcoord-dml_1_447880825398107025v0] [size=58] [limit=3592] [memorySize=249230]
[2024/04/16 15:12:28.029 +00:00] [INFO] [datanode/channel_meta.go:608] ["updated segment num row"] [segmentID=448803498411568777] [addedNumRows=7] [numRowsSum=2639]
[2024/04/16 15:12:28.030 +00:00] [INFO] [datanode/channel_meta.go:634] ["updated segment memorySize"] [segmentID=448803498411568777] [memorySize=32565]
[2024/04/16 15:12:28.030 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:286] ["segment buffer status"] [no.=0] [segmentID=448803498411568777] [channel=by-dev-rootcoord-dml_0_447880825398106991v0] [size=58] [limit=20069] [memorySize=32565]
[2024/04/16 15:12:28.029 +00:00] [INFO] [pipeline/insert_node.go:80] ["pipeline fetch insert msg"] [collectionID=447880825398106991] [segmentID=448803498411568777] [insertRowNum=7] [timestampMin=449126163460653058] [timestampMax=449126163460653058]
[2024/04/16 15:12:28.029 +00:00] [INFO] [pipeline/insert_node.go:80] ["pipeline fetch insert msg"] [collectionID=447880825398107025] [segmentID=448803498411599794] [insertRowNum=7] [timestampMin=449126163460653060] [timestampMax=449126163460653060]
[2024/04/16 15:12:28.433 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
[2024/04/16 15:12:29.429 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
[2024/04/16 15:12:30.428 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
[2024/04/16 15:12:30.911 +00:00] [INFO] [proxy/meta_cache.go:900] ["expire all shard leader cache"] [database=default] [collections="[document_texts,vectors_ml_e5_large,ai_helpdesk,documents_EN,documents_DE,documents]"]
[2024/04/16 15:12:31.429 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
[2024/04/16 15:12:32.428 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
I20240416 15:12:34.061237 44042 time_recorder.cc:49] [KNOWHERE][PrintTimeRecord][milvus] Build index: done (68980.911921 ms)
[2024/04/16 15:12:34.126 +00:00] [WARN] [tso/tso.go:178] ["clock offset is huge, check network latency and clock skew"] [jet-lag=409.380249ms] [prev-physical=2024/04/16 15:12:33.550 +00:00] [now=2024/04/16 15:12:33.959 +00:00]
[2024/04/16 15:12:34.064 +00:00] [INFO] [proxy/meta_cache.go:900] ["expire all shard leader cache"] [database=default] [collections="[vectors_ml_e5_large,ai_helpdesk,documents_EN,documents_DE,documents,document_texts]"]
[2024/04/16 15:12:33.529 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
[2024/04/16 15:12:34.317 +00:00] [INFO] [indexnode/task.go:363] ["Successfully build index"] [buildID=448803498411600556] [Collection=447880825398107025] [SegmentID=448803498404092843] [currentIndexVersion=1]
I20240416 15:12:34.334169 44059 ivf.cc:699] [KNOWHERE][Serialize][milvus] request version 1
[2024/04/16 15:12:34.412 +00:00] [WARN] [rootcoord/timeticksync.go:313] ["rootcoord send tt to all channels too slowly"] [chanNum=6] [span=213]
[2024/04/16 15:12:34.431 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
I20240416 15:12:35.336944 44059 ivf.cc:705] [KNOWHERE][Serialize][milvus] write IVF_FLAT, file size 535913399
[2024/04/16 15:12:35.429 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]
[2024/04/16 15:12:35.650 +00:00] [INFO] [datacoord/index_service.go:513] ["receive DescribeIndex request"] [traceID=a6ed6e8fe9d595751413ca3f6f2c66c9] [collectionID=447880825398106991] [indexName=] [timestamp=0]
[2024/04/16 15:12:35.651 +00:00] [INFO] [datacoord/index_service.go:449] ["completeIndexInfo success"] [collectionID=447880825398106991] [indexID=447880825398106997] [totalRows=271127] [indexRows=271127] [pendingIndexRows=0] [state=Finished] [failReason=]
[2024/04/16 15:12:35.651 +00:00] [INFO] [datacoord/index_service.go:560] ["DescribeIndex success"] [traceID=a6ed6e8fe9d595751413ca3f6f2c66c9] [collectionID=447880825398106991] [indexName=]
[2024/04/16 15:12:35.651 +00:00] [INFO] [datacoord/index_service.go:513] ["receive DescribeIndex request"] [traceID=8db95ba8a613a68680eb1e4b639b0265] [collectionID=444725728207049774] [indexName=] [timestamp=0]
[2024/04/16 15:12:35.651 +00:00] [INFO] [datacoord/index_service.go:449] ["completeIndexInfo success"] [collectionID=444725728207049774] [indexID=444725728207049778] [totalRows=9069] [indexRows=9069] [pendingIndexRows=0] [state=Finished] [failReason=]
[2024/04/16 15:12:35.651 +00:00] [INFO] [datacoord/index_service.go:560] ["DescribeIndex success"] [traceID=8db95ba8a613a68680eb1e4b639b0265] [collectionID=444725728207049774] [indexName=]
[2024/04/16 15:12:35.652 +00:00] [INFO] [datacoord/index_service.go:513] ["receive DescribeIndex request"] [traceID=e87b526eae44760460084b5d020edbd5] [collectionID=444725728210909993] [indexName=] [timestamp=0]
[2024/04/16 15:12:35.652 +00:00] [INFO] [datacoord/index_service.go:449] ["completeIndexInfo success"] [collectionID=444725728210909993] [indexID=447042714307813983] [totalRows=530271] [indexRows=530271] [pendingIndexRows=0] [state=Finished] [failReason=]
[2024/04/16 15:12:35.652 +00:00] [INFO] [datacoord/index_service.go:560] ["DescribeIndex success"] [traceID=e87b526eae44760460084b5d020edbd5] [collectionID=444725728210909993] [indexName=]
[2024/04/16 15:12:35.653 +00:00] [INFO] [querynodev2/services.go:1366] ["sync action"] [traceID=ce3f5e08d8dc35a64a21fb7e269ceb7a] [collectionID=447880825398106991] [channel=by-dev-rootcoord-dml_0_447880825398106991v0] [currentNodeID=12] [Action=UpdateVersion] [TargetVersion=1713280345654142257]
[2024/04/16 15:12:35.656 +00:00] [INFO] [datacoord/index_service.go:513] ["receive DescribeIndex request"] [traceID=8357fa31384ee475a16d696f970ba8fa] [collectionID=447880825398107025] [indexName=] [timestamp=0]
[2024/04/16 15:12:35.656 +00:00] [INFO] [datacoord/index_service.go:449] ["completeIndexInfo success"] [collectionID=447880825398107025] [indexID=447880825398107030] [totalRows=272665] [indexRows=272665] [pendingIndexRows=129433] [state=Finished] [failReason=]
[2024/04/16 15:12:35.656 +00:00] [INFO] [datacoord/index_service.go:560] ["DescribeIndex success"] [traceID=8357fa31384ee475a16d696f970ba8fa] [collectionID=447880825398107025] [indexName=]
[2024/04/16 15:12:35.657 +00:00] [INFO] [datacoord/index_service.go:513] ["receive DescribeIndex request"] [traceID=7ddd149a134895145d1b5b516d11889b] [collectionID=444725728207049736] [indexName=] [timestamp=0]
[2024/04/16 15:12:35.657 +00:00] [INFO] [datacoord/index_service.go:449] ["completeIndexInfo success"] [collectionID=444725728207049736] [indexID=444725728207049740] [totalRows=9069] [indexRows=9069] [pendingIndexRows=0] [state=Finished] [failReason=]
[2024/04/16 15:12:35.657 +00:00] [INFO] [datacoord/index_service.go:560] ["DescribeIndex success"] [traceID=7ddd149a134895145d1b5b516d11889b] [collectionID=444725728207049736] [indexName=]
[2024/04/16 15:12:35.658 +00:00] [INFO] [delegator/distribution.go:294] ["Update readable segment version"] [oldVersion=1713280335666636444] [newVersion=1713280345654142257] [growingSegmentNum=1] [sealedSegmentNum=1]
[2024/04/16 15:12:35.659 +00:00] [INFO] [querynodev2/services.go:1366] ["sync action"] [traceID=dd2a282ee7fd9f9935905aa6df15e23c] [collectionID=444725728210909993] [channel=by-dev-rootcoord-dml_6_444725728210909993v0] [currentNodeID=12] [Action=UpdateVersion] [TargetVersion=1713280345654482140]
[2024/04/16 15:12:35.659 +00:00] [INFO] [delegator/distribution.go:294] ["Update readable segment version"] [oldVersion=1713280335661133232] [newVersion=1713280345654482140] [growingSegmentNum=0] [sealedSegmentNum=6]
[2024/04/16 15:12:35.659 +00:00] [INFO] [observers/target_observer.go:473] ["observer trigger update current target"] [collectionID=444725728210909993]
[2024/04/16 15:12:35.661 +00:00] [INFO] [querynodev2/services.go:1366] ["sync action"] [traceID=687ef4159ab08306e47534eb5878deb4] [collectionID=444725728207049774] [channel=by-dev-rootcoord-dml_3_444725728207049774v0] [currentNodeID=12] [Action=UpdateVersion] [TargetVersion=1713280345654410662]
[2024/04/16 15:12:35.661 +00:00] [INFO] [delegator/distribution.go:294] ["Update readable segment version"] [oldVersion=1713280335664399979] [newVersion=1713280345654410662] [growingSegmentNum=0] [sealedSegmentNum=2]
[2024/04/16 15:12:35.661 +00:00] [INFO] [querynodev2/services.go:1366] ["sync action"] [traceID=52ced16f6607061ba16479c8d10c9287] [collectionID=444725728207049736] [channel=by-dev-rootcoord-dml_2_444725728207049736v0] [currentNodeID=12] [Action=UpdateVersion] [TargetVersion=1713280345654270124]
[2024/04/16 15:12:35.661 +00:00] [INFO] [delegator/distribution.go:294] ["Update readable segment version"] [oldVersion=1713280335661490246] [newVersion=1713280345654270124] [growingSegmentNum=0] [sealedSegmentNum=2]
[2024/04/16 15:12:35.662 +00:00] [INFO] [querynodev2/services.go:1366] ["sync action"] [traceID=82f8572ea738a53dbf84d36d12d47ac2] [collectionID=447880825398107025] [channel=by-dev-rootcoord-dml_1_447880825398107025v0] [currentNodeID=12] [Action=UpdateVersion] [TargetVersion=1713280345654902583]
[2024/04/16 15:12:35.663 +00:00] [INFO] [delegator/distribution.go:294] ["Update readable segment version"] [oldVersion=1713280335660187976] [newVersion=1713280345654902583] [growingSegmentNum=0] [sealedSegmentNum=3]
[2024/04/16 15:12:35.663 +00:00] [INFO] [datacoord/index_service.go:513] ["receive DescribeIndex request"] [traceID=3cb69afe91255ec8fa12b952378caa84] [collectionID=447042714314267305] [indexName=] [timestamp=0]
[2024/04/16 15:12:35.663 +00:00] [INFO] [datacoord/index_service.go:449] ["completeIndexInfo success"] [collectionID=447042714314267305] [indexID=447042714314267311] [totalRows=35] [indexRows=35] [pendingIndexRows=0] [state=Finished] [failReason=]
[2024/04/16 15:12:35.663 +00:00] [INFO] [datacoord/index_service.go:560] ["DescribeIndex success"] [traceID=3cb69afe91255ec8fa12b952378caa84] [collectionID=447042714314267305] [indexName=]
[2024/04/16 15:12:35.663 +00:00] [INFO] [datacoord/services.go:757] ["get recovery info request received"] [traceID=e3333646adb3dd4c4438d5c44e000bb1] [collectionID=447880825398107025] [partitionIDs="[]"]
[2024/04/16 15:12:35.664 +00:00] [INFO] [querynodev2/services.go:1366] ["sync action"] [traceID=c7d0b98e4ce4af35cb93b4e6b4c495fc] [collectionID=447042714314267305] [channel=by-dev-rootcoord-dml_7_447042714314267305v0] [currentNodeID=12] [Action=UpdateVersion] [TargetVersion=1713280345653580932]
[2024/04/16 15:12:35.664 +00:00] [INFO] [delegator/distribution.go:294] ["Update readable segment version"] [oldVersion=1713280335666048447] [newVersion=1713280345653580932] [growingSegmentNum=0] [sealedSegmentNum=1]
[2024/04/16 15:12:35.664 +00:00] [INFO] [datacoord/services.go:757] ["get recovery info request received"] [traceID=0327d0c765063c2bd96f043c29dd8dc2] [collectionID=444725728207049774] [partitionIDs="[]"]
[2024/04/16 15:12:35.664 +00:00] [INFO] [datacoord/services.go:757] ["get recovery info request received"] [traceID=a938811b88cdf6e58940ff02eb78ad3d] [collectionID=447880825398106991] [partitionIDs="[]"]
[2024/04/16 15:12:35.664 +00:00] [INFO] [datacoord/services.go:757] ["get recovery info request received"] [traceID=92a0c28e116f5f68b4550d993bae0417] [collectionID=444725728207049736] [partitionIDs="[]"]
[2024/04/16 15:12:35.665 +00:00] [INFO] [datacoord/handler.go:116] [GetQueryVChanPositions] [collectionID=444725728207049774] [channel=by-dev-rootcoord-dml_3_444725728207049774v0] [numOfSegments=2] ["indexed segment"=2]
[2024/04/16 15:12:35.665 +00:00] [INFO] [datacoord/handler.go:299] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_3_444725728207049774v0] [posTs=449126159240134657] [posTime=2024/04/16 15:12:11.574 +00:00]
[2024/04/16 15:12:35.665 +00:00] [INFO] [datacoord/services.go:772] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=0327d0c765063c2bd96f043c29dd8dc2] [collectionID=444725728207049774] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_3_444725728207049774v0] ["# of unflushed segments"=0] ["# of flushed segments"=2] ["# of dropped segments"=0] ["# of indexed segments"=0]
[2024/04/16 15:12:35.665 +00:00] [INFO] [datacoord/handler.go:116] [GetQueryVChanPositions] [collectionID=444725728207049736] [channel=by-dev-rootcoord-dml_2_444725728207049736v0] [numOfSegments=2] ["indexed segment"=2]
[2024/04/16 15:12:35.665 +00:00] [INFO] [datacoord/handler.go:299] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_2_444725728207049736v0] [posTs=449126159240134657] [posTime=2024/04/16 15:12:11.574 +00:00]
[2024/04/16 15:12:35.665 +00:00] [INFO] [datacoord/services.go:772] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=92a0c28e116f5f68b4550d993bae0417] [collectionID=444725728207049736] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_2_444725728207049736v0] ["# of unflushed segments"=0] ["# of flushed segments"=2] ["# of dropped segments"=0] ["# of indexed segments"=0]
[2024/04/16 15:12:35.665 +00:00] [INFO] [datacoord/handler.go:116] [GetQueryVChanPositions] [collectionID=447880825398107025] [channel=by-dev-rootcoord-dml_1_447880825398107025v0] [numOfSegments=6] ["indexed segment"=4]
[2024/04/16 15:12:35.666 +00:00] [INFO] [datacoord/handler.go:299] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_1_447880825398107025v0] [posTs=449126111372640257] [posTime=2024/04/16 15:09:08.974 +00:00]
[2024/04/16 15:12:35.666 +00:00] [INFO] [datacoord/services.go:772] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=e3333646adb3dd4c4438d5c44e000bb1] [collectionID=447880825398107025] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_1_447880825398107025v0] ["# of unflushed segments"=0] ["# of flushed segments"=3] ["# of dropped segments"=1] ["# of indexed segments"=0]
[2024/04/16 15:12:35.665 +00:00] [INFO] [datacoord/services.go:757] ["get recovery info request received"] [traceID=74cfc868a3df9e7af83dfed1c8ac0418] [collectionID=444725728210909993] [partitionIDs="[]"]
[2024/04/16 15:12:35.666 +00:00] [INFO] [datacoord/handler.go:116] [GetQueryVChanPositions] [collectionID=444725728210909993] [channel=by-dev-rootcoord-dml_6_444725728210909993v0] [numOfSegments=6] ["indexed segment"=6]
[2024/04/16 15:12:35.666 +00:00] [INFO] [datacoord/handler.go:299] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_6_444725728210909993v0] [posTs=449126159240134657] [posTime=2024/04/16 15:12:11.574 +00:00]
[2024/04/16 15:12:35.666 +00:00] [INFO] [datacoord/services.go:772] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=74cfc868a3df9e7af83dfed1c8ac0418] [collectionID=444725728210909993] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_6_444725728210909993v0] ["# of unflushed segments"=0] ["# of flushed segments"=6] ["# of dropped segments"=0] ["# of indexed segments"=0]
[2024/04/16 15:12:35.665 +00:00] [INFO] [datacoord/handler.go:116] [GetQueryVChanPositions] [collectionID=447880825398106991] [channel=by-dev-rootcoord-dml_0_447880825398106991v0] [numOfSegments=2] ["indexed segment"=1]
[2024/04/16 15:12:35.666 +00:00] [INFO] [datacoord/handler.go:299] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_0_447880825398106991v0] [posTs=449126111372640257] [posTime=2024/04/16 15:09:08.974 +00:00]
[2024/04/16 15:12:35.666 +00:00] [INFO] [datacoord/services.go:772] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=a938811b88cdf6e58940ff02eb78ad3d] [collectionID=447880825398106991] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_0_447880825398106991v0] ["# of unflushed segments"=1] ["# of flushed segments"=1] ["# of dropped segments"=0] ["# of indexed segments"=0]
[2024/04/16 15:12:35.664 +00:00] [INFO] [datacoord/services.go:757] ["get recovery info request received"] [traceID=b3f2a02d01619a28f732933873048d7f] [collectionID=447042714314267305] [partitionIDs="[]"]
[2024/04/16 15:12:35.666 +00:00] [INFO] [datacoord/handler.go:116] [GetQueryVChanPositions] [collectionID=447042714314267305] [channel=by-dev-rootcoord-dml_7_447042714314267305v0] [numOfSegments=1] ["indexed segment"=1]
[2024/04/16 15:12:35.666 +00:00] [INFO] [datacoord/handler.go:299] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_7_447042714314267305v0] [posTs=449126153420537857] [posTime=2024/04/16 15:11:49.374 +00:00]
[2024/04/16 15:12:35.666 +00:00] [INFO] [datacoord/services.go:772] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=b3f2a02d01619a28f732933873048d7f] [collectionID=447042714314267305] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_7_447042714314267305v0] ["# of unflushed segments"=0] ["# of flushed segments"=1] ["# of dropped segments"=0] ["# of indexed segments"=0]
[2024/04/16 15:12:49.432 +00:00] [INFO] [proxy/meta_cache.go:900] ["expire all shard leader cache"] [database=default] [collections="[documents_EN,documents_DE,documents,document_texts,vectors_ml_e5_large,ai_helpdesk]"]
[2024/04/16 15:12:49.643 +00:00] [INFO] [proxy/meta_cache.go:900] ["expire all shard leader cache"] [database=default] [collections="[documents,document_texts,vectors_ml_e5_large,ai_helpdesk,documents_EN,documents_DE]"]
[2024/04/16 15:12:49.558 +00:00] [INFO] [observers/target_observer.go:450] ["Update readable segment version"] [collectionID=447880825398106991] [channelName=by-dev-rootcoord-dml_0_447880825398106991v0] [nodeID=12] [oldVersion=1713280345654142257] [newVersion=1713280355661739174]
[2024/04/16 15:12:49.411 +00:00] [INFO] [datacoord/index_builder.go:186] ["index builder task schedule"] ["task num"=1]

Anything else?

No response

could you try to upgrade to 2.3.13 an see? And did you see the error of etcd session timeout or something?

@mihailyanchev
Copy link
Author

@yanliang567 Thank you for the quick response. Sorry, it took me some time to get the birdwatcher backup. See attached:
bw_etcd_ALL.240418-074900.bak.gz

@xiaofan-luan there were no errors with respect to etcd session timeout. The only errors in the whole milvus-standalone log, were right after the start container:

[2024/04/02 09:18:00.965 +00:00] [ERROR] [grpcclient/client.go:462] ["retry func failed"] ["retry time"=0] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:462\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:548\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:564\ngithub.com/milvus-io/milvus/internal/distributed/datacoord/client.wrapGrpcCall[...]\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:101\ngithub.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:115\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:37\ngithub.com/milvus-io/milvus/pkg/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:44\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:61\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:82\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:187\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:100\ngithub.com/milvus-io/milvus/cmd/components.(*QueryCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/query_coord.go:53\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:115"]
[2024/04/02 09:18:01.220 +00:00] [ERROR] [sessionutil/session_util.go:475] ["retry func failed"] ["retry time"=0] [error="function CompareAndSwap error for compare is false for key: datacoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:475\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:295\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Register\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:254\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:205\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:254\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:50\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:115"]

However, these appeared only a couple of times and this was 2 months ago when the container was originally started. There have been no errors since.

Otherwise, from your comments I have the following takeaway:

  • Will upgrade to milvus 2.3.13
  • Will increase compute resources (I can even decrease them initially to see if these crashes occur more often)
  • I do separate DELETE and INSERT operations instead of a single UPSERT operation, because the number of records I insert after the delete might be less or more and it seemed easier for implementing at the time. However, now I realise this might be a problem and I might switch to a different strategy of purging all records that need to be deleted once a day.

Let me know if you have any other ideas and thank you!

@yanliang567
Copy link
Contributor

@mihailyanchev the meta in etcd backup looks ok. one quick question: the backup is about milvus 2.3.4 or milvus 2.3.13(after upgrading?), did this milvus instance upgrade from 2.2.x? I'm asking because I saw 2 different index versions in the meta.

@yanliang567
Copy link
Contributor

please keep us posted as you upgrade to mivlus 2.3.13 and increase/decrease the resource. thanks

@mihailyanchev
Copy link
Author

@mihailyanchev the meta in etcd backup looks ok. one quick question: the backup is about milvus 2.3.4 or milvus 2.3.13(after upgrading?), did this milvus instance upgrade from 2.2.x? I'm asking because I saw 2 different index versions in the meta.

I have not performed the upgrade to 2.3.13 yet, so the current version is still 2.3.4. However, around 2 months ago we upgraded from 2.2.x. I followed this guide.

I will let you know how my experimentation goes. Thanks for your help again!

@stale Stale
Copy link

stale bot commented May 18, 2024

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

@stale stale bot added the stale indicates no udpates for 30 days label May 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mihailyanchev mihailyanchev

@yanliang567 yanliang567

Labels
kind/bug Issues or changes related a bug stale indicates no udpates for 30 days triage/needs-information Indicates an issue needs more information in order to work on it.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mihailyanchev @yanliang567 @xiaofan-luan