heal: Persist MRF queue in the disk during shutdown

[vadmeste]

Community Contribution License

All community contributions in this pull request are licensed to the project maintainers
under the terms of the Apache 2 license.
By creating this pull request I represent that I have the right to license the
contributions to the project maintainers under the Apache 2 license.

Description

Motivation and Context

How to test this PR?

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Optimization (provides speedup with no functional changes)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• Fixes a regression (If yes, please add commit-id or PR # here)
• Unit tests added/updated
• Internal documentation updated
• Create a documentation update request here

[harshavardhana]

Looks reordering caused some issue in healing tests PTAL @vadmeste

[harshavardhana]

@vadmeste please rebase this PR with master.

[bh4t]

@vadmeste PTAL when you get a chance.

[krisis]

This select statement doesn't guarantee that this method won't send op over m.opCh. This is because once mrfState.shutdown() is called, both case <-m.closing and case m.opCh <-op may be ready and select statement picks one among them at random.

A select blocks until one of its cases can run, then it executes that case. It chooses one at random if multiple are ready.
ref: https://go.dev/tour/concurrency/5

Instead we should do like this,

select {
case <-m.closing:
      m.closeOnce.Do(func() {close(m.opCh))}
default:
      select {
      case m.opCh <- op:
      default:
      }
}