refine update partition stats mechanism(#32446) #32768
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: MrPresent-Han The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
sre-ci-robot
added
size/L
MrPresent-Han
force-pushed
the
refine-part-status-lru-dev-v1
branch
from
c6a093a
to
61dbb3c
Compare
|
|
||
| req := &querypb.SyncDistributionRequest{ | ||
| Base: commonpbutil.NewMsgBase( | ||
| commonpbutil.WithMsgType(commonpb.MsgType_LoadSegments), |
There was a problem hiding this comment.
is thisMsgType_LoadSegments right?
| } | ||
| log := log.Ctx(ctx) | ||
| for partID, newVersion := range partStatsVersions { | ||
| curStats, exist := sd.partitionStats[partID] |
Signed-off-by: MrPresent-Han <[email protected]>
MrPresent-Han
force-pushed
the
refine-part-status-lru-dev-v1
branch
from
61dbb3c
to
aa342d0
Compare
sre-ci-robot
added
size/XL
| @@ -206,14 +207,35 @@ func (h *ServerHandler) GetQueryVChanPositions(channel RWChannel, partitionIDs . | |||
| // unindexed is flushed segments as well | |||
| indexedIDs.Insert(unIndexedIDs.Collect()...) | |||
|
|
|||
| // update partition stats versions | |||
There was a problem hiding this comment.
when there are too many partitions and multiple versions of partition stats are avaiable to single partition, this part can consume loads of cpu cycles, no mentioning it includes string prefix compare
| log.Info("update partition stats versions") | ||
| sd.loadPartitionStats(ctx, partVersions) | ||
| } | ||
|
|
There was a problem hiding this comment.
Rlock protect get GetPartitionStatsVersions.
WLock protection is inside loadPartitionStats(ctx, partVersions)
| func() { | ||
| sd.partitionStatsMut.RLock() | ||
| defer sd.partitionStatsMut.RUnlock() | ||
| partStats, exists = sd.partitionStats[partID] |
There was a problem hiding this comment.
only use mutex in the last step when upgrading the underlying partition stats in the memory to avoid blocking search process for slow s3-read process
related: #32446