[Feature]: Allow writes during a node outage

[Zorlin]

Contact Details

No response

Is there an existing issue for this?

• I have searched all the existing issues

Is your feature request related to a problem? Please describe.

I would like to have CubeFS survive taking datanodes and metanodes offline.

Right now if I stop a datanode or have a physical failure, my cluster can no longer receive writes even on a 3-replica volume with 5 available datanodes.

Read workloads continue to work fine, but this stops me from doing certain kinds of work during the read-only mode.

Describe the solution you'd like.

We should implement a file/chunk versioning system combined with CRDTs (https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type) to allow us to continue writing data when a DP shard is offline, and then update that shard when it returns.

Describe an alternate solution.

No response

Anything else? (Additional Context)

Other filesystems such as Ceph, MooseFS do things like this to allow for writing even during major node outages.

[leonrayang]

@Zorlin I don't know if you have any practical experience, but the problem should not exist. If one out of five nodes is faulty, writing can still continue without affecting users. The data will be appended to a new node, and modifications will be made to the remaining two replicas, which have a leader, so everything is also okay.

[Zorlin]

@Zorlin I don't know if you have any practical experience, but the problem should not exist. If one out of five nodes is faulty, writing can still continue without affecting users. The data will be appended to a new node, and modifications will be made to the remaining two replicas, which have a leader, so everything is also okay.

Hi, agreed it shouldn't be a problem, but in my cluster with 5 datanodes and 5 metanodes, when I am writing to the filesystem and I take a node offline, everything "goes haywire" and I cannot write again until I put that node back online.

[NaturalSelect]

@Zorlin I don't know if you have any practical experience, but the problem should not exist. If one out of five nodes is faulty, writing can still continue without affecting users. The data will be appended to a new node, and modifications will be made to the remaining two replicas, which have a leader, so everything is also okay.

Hi, agreed it shouldn't be a problem, but in my cluster with 5 datanodes and 5 metanodes, when I am writing to the filesystem and I take a node offline, everything "goes haywire" and I cannot write again until I put that node back online.

How many data partitions does your volume have?