CompleteMultiPartUpload java.lang.NullPointerException on OM restart

[zorel]

Hello.

I got a this error on OM restart. I suppose it's a messed up multipart upload, after we did tests with Spark+S3 . It's a test env so I can erase everything with no problem, but still...

ERROR ratis.OzoneManagerStateMachine: Terminating with exit status 1: Request cmdType: CompleteMultiPartUpload
< here a json >
java.lang.NullPointerException: Cannot invoke "org.apache.hadoop.ozone.om.helpers.OmKeyInfo.getKeyName()" because "dbOpenKeyInfo" is null
	at org.apache.hadoop.ozone.om.request.s3.multipart.S3MultipartUploadCompleteRequest.getOmKeyInfo(S3MultipartUploadCompleteRequest.java:385)
	at org.apache.hadoop.ozone.om.request.s3.multipart.S3MultipartUploadCompleteRequest.validateAndUpdateCache(S3MultipartUploadCompleteRequest.java:204)
	at org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.lambda$0(OzoneManagerRequestHandler.java:402)
	at org.apache.hadoop.util.MetricUtil.captureLatencyNs(MetricUtil.java:39)
	at org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequest(OzoneManagerRequestHandler.java:398)
	at org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.runCommand(OzoneManagerStateMachine.java:587)
	at org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$1(OzoneManagerStateMachine.java:375)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

I'll dig with ozone debug, is there anything I can do?

[adoroszlai]

@zorel #6496 may be fixing this bug.

[zorel]

Thanks. It will land in 1.4.1? Is there a workaround in between?

[adoroszlai]

It will land in 1.4.1?

I hope so. If not in 1.4.1, maybe 1.4.2.

Is there a workaround in between?

If you are using S3A, either of these settings should work around the bug:

• fs.s3a.committer.name=magic
• fs.s3a.committer.staging.conflict-mode=append

[zorel]

In fact, I have the error on OM startup. The OM does not start anymore with this message.
It seems there is no way to modifiy the rocksdb data without writing some code, and as it was a test, I deleted everything and start over from scratch.

[adoroszlai]

You are right, sorry, the committer settings would not fix broken entries already in OM DB. They could be useful preventing it from happening again.