[BUG] IStateSerializer Concurrency Issues (Protobuf)

[RagtimeWilly]

Describe the bug
We introduced an IStateSerializer (as per docs 1 2) which utilizes protobuf (protobuf-net) as the serializer.

This appeared to work fine but after running for a few weeks we started to see issues where key/values were getting mixed up, i.e, a read for key A was returning a value B.

A concurrency issue with the serializer during the writes is our suspicion - but I cannot identify an issue with the code. We rolled back the change and have seen no occurrences since.

Would anybody be able to advise whether there is an issue with the implementation or if this is a bug with the framework?

public abstract class ProtobufStateSerializer<T> : IStateSerializer<T>
{
    // Called from constructor of the StatefulService.
    public void RegisterWith(IReliableStateManager stateManager)
    {
        if (!stateManager.TryAddStateSerializer(this))
        {
            throw new Exception($"Failed to set {typeof(T).Name} custom serializer");
        }
    }

    public T Read(BinaryReader binaryReader)
    {
        var bytes = ReadAllBytes(binaryReader);
        
        return ProtoBuf.Serializer.Deserialize<T>(bytes);
    }

    public void Write(T value, BinaryWriter binaryWriter)
    {
        byte[] result;

        using (var stream = new MemoryStream())
        {
            ProtoBuf.Serializer.Serialize(stream, value);
            result = stream.ToArray();
        }

        binaryWriter.Write(result);
    }
    
    // Read overload for differential de-serialization
    T IStateSerializer<T>.Read(T baseValue, BinaryReader reader)
    {
        return ((IStateSerializer<T>)this).Read(reader);
    }

    // Write overload for differential serialization
    void IStateSerializer<T>.Write(T baseValue, T newValue, BinaryWriter writer)
    {
        ((IStateSerializer<T>)this).Write(newValue, writer);
    }

    private static ReadOnlySpan<byte> ReadAllBytes(BinaryReader reader)
    {
        const int bufferSize = 4096;
        
        using var ms = new MemoryStream();
        
        var buffer = new byte[bufferSize];
        
        int count;
        
        while ((count = reader.Read(buffer, 0, buffer.Length)) != 0)
            ms.Write(buffer, 0, count);
        
        return ms.ToArray();
    }
}

// Then implemented per type as:

public class SomeTypeSerializer : ProtobufStateSerializer<SomeType>
{
}

Area/Component:
Reliable services/SDK

To Reproduce
It is difficult to reproduce as it only manifested sporadically and after a number of weeks.

Expected behavior
No isolation issues when using custom serializer

Observed behavior:
See Description.

Service Fabric Runtime Version:
6.1.1436

Environment:

• Azure
• OS: Windows Server 2022
• SF 10.0.1816.9590

---

Assignees: /cc @microsoft/service-fabric-triage