2024 March APAC Community Sync

[ChenSammi]

Attender:Hao, Mingyu, Yiyang, Xi, Bin, Jianghua, Hualong, Yuanben, Conway, Chungen, Wei-Chiu, Sammi

1. DiDi
   a. Bad replicas are not timely detected and reported in the system so that healthy replicas are deleted by RM. Some data is lost due to this. Suggest turning on Disk Scanner by turning on "hdds.container.scrub.enabled".
   b. Two consecutive tasks are scheduled to replicate the same container on the same DN. HDDS-9661 has fixed this issue.
   c. Plan to use high density storage servers as Ozone DN. Wei-Chiu shares a document about the last test experience of Ozone on high density storage.
2. Shopee
   a. When DN is dead or decommissioned, DN info is still held in SCM memory. After SCM restarts, those DNs info will be gone. But Recon still shows these DNs on the UI.
   b. Want to know the different type buckets and their supportability in each Ozone access interface(s3g, ofs, cli). https://blog.cloudera.com/apache-ozone-a-multi-protocol-aware-storage-system/
3. Qihoo
   a. multipart key info serialization and deserialization overhead is very high for big objects(100G). Evaluate to add a new table for multipart key info. Will create a JIRA for this later.
   b. Observed that many objects in OM are serialized and deserialized multiple times, which add the pressure to GC.
   c. Per user throughput throttle to avoid most of the throughput being used by one user. FairCallQueue is suggested.

[ChenSammi]

15th March
Attender:Yiyang, Xi, Hao, Mingyu, Shawn, HongBing, Jianghua, Hualong, Yuanben, Conway, Chungen, Jeff, Wei-Chiu, Sammi

1. Cloudera
   a. The issue found in HDDS-9342 has potential to lose data. It's fixed by HDDS-9876. Suggest to community partners to upgrade to 1.4.0 or apply the patch HDDS-9876.
   b. Ratis streaming performance data sharing
   c. Hbase startus update
2. Qihoo
   a. One volume is marked as failure due to long RocksDB compaction time. HDDS-8784, will submit improvement patch later.
3. Shoppe
   a. Namespace quota discussion. Currently if the space quota is less than 1 default block size, file/key creation request will fail. It's the expected behavior.
   b. Client side block size overwrite, allow client to specify different block size, other than use the default block size.
4. DiDi
   a. OM long GC(>60s) with 64GB heap size for near 2 billion file/objects. Need more info (jmap) to continue the investigation.
   b. Test JuiceFS on Ozone in the plan.

[ChenSammi]

22th March
Attender:Ivan, Conway, Siddhant, Tanvi, Sumit, Wei-Chiu, Sammi
Agenda:

1. Shopee
   a. Investigating OM federation solution.
   b. Investigating S3 object lifecycle management. HDDS-8342.
   c. Plan to test OM follower read
2. Cloudera
   a. Discussed the star mode and pipeline mode of Ratis Streaming("ozone.client.datastream.pipeline.mode")

[ChenSammi]

29th March
Attender: Hongbing, Yuanben, Conway, Hualong, Jianghua, Xi, Hao, Minyu, Shawn, Lee Chungen, Sammi
Agenda:
DiDi
- Reported HDDS-10608, Recon can't get full key when using Recon API
- Need HDFS fsck similar function for issue investigation
- Have finished follower read feature function test
Qihoo
- There is a need to migrate SCM HA instances and OM HA instances to new servers. Drafting a migration plan to minimum the service unavailability time duration as small as possible.
- Opened JIRA HDDS-10611, Optimization of gc pressure on om due to handling of mpu large files. Will have a design document attached later.
- Discussed if it's possible for OMClientRequest#preExecute to avoid allocate new OMRequest on each call, to reduce the GC pressure.

[errose28]

There is a need to migrate SCM HA instances and OM HA instances to new servers. Drafting a migration plan to minimum the service unavailability time duration as small as possible.

This would be great to have in the docs on the new website. I'm not sure if HDDS-10451 is planning to track this but please file a new Jira under HDDS-9859 if this one does not match up with this task. Even adding an outline of the proposal to Jira can help us get started converting it to an actual docs page.

[ivandika3]

Our cluster did a similar migration for OM and SCM HA last year. My intent for HDDS-10451 is to highlight the transfer leadership tool and how to use it. One use case that might is during a rolling OM / SCM HA release, where we transfer the leadership away from the OM / SCM server to be restarted so that there is minimal downtime.

For HA migration, I think we can add another docs page for this, since this need to include possible decommissions of old OM / SCM servers that is out of the scope of transfer leadership.

I'll try to make some time to add these documentations.