Ozone 1.4.0 got "org.eclipse.jetty.io.EofException: Early EOF" error for uploading large objects

[Lucas12138]

Hi,

we recently upgraded from ozone 1.3.0 to 1.4.0 and enabled erasure coding.
After this, we did some testing. For smaller objects (less than 100GB), usually it works fine.
For large objects, e.g. 1TB, we got the following errors after uploading for around 30min (at the speed of 66492914 bytes per second).

It had around 400 records of success logs like:

2024-05-06 18:04:33,849 | INFO  | S3GAudit | user= XXX | ip= XXX | op=CREATE_MULTIPART_KEY {bucket=[XXXXX], path=[XXXXX/XXXXX.mov], uploadId=[76219e9a-9530-4b81-a275-d80c1489758c-112395337692415491], partNumber=[423]} | ret=SUCCESS | perf={sizeByte=293095115, metaLatencyMs=3, opLatencyMs=4405}

then went into

2024-05-06 18:04:36,068 | ERROR | S3GAudit | user=XXX | ip=XXX | op=CREATE_MULTIPART_KEY {bucket=[XXXXX], path=[XXXXX/XXXXX.mov], uploadId=[76219e9a-9530-4b81-a275-d80c1489758c-112395337692415491], partNumber=[424]} | ret=FAILURE | org.eclipse.jetty.io.EofException: Early EOF
at org.eclipse.jetty.server.HttpInput$3.getError(HttpInput.java:1207)
at org.eclipse.jetty.server.HttpInput$3.noContent(HttpInput.java:1195)

and then the upload was stopped.

Can you help see what may be the cause of this? Thank you

[ivandika3]

@Lucas12138 In my experience S3G Jetty EofException was usually thrown either due to 1) S3 client cancelling the request (e.g. client ctrl-c while the S3 client is in the middle of uploading) or 2) Network instability during upload that disrupt the uploading process.

May I know how big is each part? One way to reduce the likelihood is to reduce the size of the parts in the multipart uploads.

Since it is a multipart upload, whenever you find this issue, you can continue the upload the parts starting from the part that threw the exception (i.e. part 424 in your example) and complete it afterwards. There is no need to reupload the previous parts. However, you might need to write a custom program to complete the rest of the multipart upload parts.

[ivandika3]

Not really sure whether EC has anything to do with it. Maybe @adoroszlai or @sodonnel knows more.