S3 storage class type for using EC

[Lucas12138]

Hi,

We are using s3 clients to upload to ozone.

ozone/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/util/S3StorageType.java

Lines 35 to 36 in db2e507

	REDUCED_REDUNDANCY(ReplicationType.RATIS, ReplicationFactor.ONE),
	STANDARD(ReplicationType.RATIS, ReplicationFactor.THREE);

Although we have configurations like the following on the ozone server side

  OZONE-SITE.XML_ozone.server.default.replication.type: EC
  OZONE-SITE.XML_ozone.server.default.replication: RS-6-3-1024k

when uploading with s3 client, we saw the key is still stored as RATIS, ONE (we used the header: REDUCED_REDUNDANCY)

Is there any s3 storage class that we can use to explicitly upload as EC, RS-6-3-1024k?

[Lucas12138]

After removing this REDUCED_REDUNDANCY header, the replication is defaulted to
"replicationFactor" : "THREE",
"requiredNodes" : 3,
"replicationType" : "RATIS"

How can we use EC and RS-6-3-1024k when we use the s3 client?

[errose28]

Looks like you hit HDDS-10327, which was resolved after the 1.4.0 release. Prior to this fix, S3 gateway would default to sending Ratis 3 as the replication type if no type was provided, and this would override the server default. With the fix, S3 gateway will not specify a replication config by default, so the OM will fall back to the server side configuration you have set to EC here.

You should be able to work around the issue by setting ozone.replication and ozone.replication.type on the S3 gateway. These specify the default values the Ozone client should use if none are provided. In this case S3 gateway is the Ozone client to the Ozone Manager.

[Lucas12138]

Thanks, after adding these props on s3g, now it's getting the following error:

javax.servlet.ServletException: javax.servlet.ServletException: java.lang.IllegalArgumentException: RS-6-3-1024k is not supported for RATIS replication type
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:162)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:516)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)

I followed https://ozone.apache.org/docs/current/feature/erasurecoding.html to setup the EC for the cluster. Mostly it's adding some props for om.

Apart from this did I miss anything?

[errose28]

On the S3 gateway set ozone.replication.type=EC and ozone.replication=RS-6-3-1024k. It looks like you didn't set ozone.replication.type and it defaulted to RATIS instead of EC.

[Lucas12138]

Just checked the logs and I think there was a typo. Let me retry and see. Thank you!

[Lucas12138]

After correcting the props, the error is gone but the object is still stored as RATIS 3:

  "dataSize" : 4011465564,
  "creationTime" : "2024-05-09T22:14:01.998Z",
  "modificationTime" : "2024-05-09T22:14:01.998Z",
  "replicationConfig" : {
    "replicationFactor" : "THREE",
    "requiredNodes" : 3,
    "replicationType" : "RATIS"
  },

[Lucas12138]

Actually I saw Please note, the above configurations will be used only when client does not pass any replication config or bucket does not have any default values. from https://ozone.apache.org/docs/current/feature/erasurecoding.html

Since the s3 client is mapped to the standard storage class which is ratis three, I wonder if it's then overriding the cluster wise replication settings?