OM won't start

[zorel]

Hi.

Got those messages repeating over and over, and OM won't start. I have 3 om in the configuration, it's the first run after the om --init.
Init has been done on SCMs too, they are running and looks ok.

I tried to clean up metadata directories multiple times and start from scratch with no luck, still same behaviour.

[om1@group-5AEFCA43398D-FollowerState] INFO impl.FollowerState: om1@group-5AEFCA43398D-FollowerState: change to CANDIDATE, lastRpcElapsedTime:5034263795ns, electionTimeout:5034ms
[om1@group-5AEFCA43398D-FollowerState] INFO impl.RoleInfo: om1: shutdown om1@group-5AEFCA43398D-FollowerState
[om1@group-5AEFCA43398D-FollowerState] INFO server.RaftServer$Division: om1@group-5AEFCA43398D: changes role from  FOLLOWER to CANDIDATE at term 0 for changeToCandidate
[om1@group-5AEFCA43398D-FollowerState] INFO server.RaftServerConfigKeys: raft.server.leaderelection.pre-vote = true (default)
[om1@group-5AEFCA43398D-FollowerState] INFO impl.RoleInfo: om1: start om1@group-5AEFCA43398D-LeaderElection13
[om1@group-5AEFCA43398D-LeaderElection13] INFO impl.LeaderElection: om1@group-5AEFCA43398D-LeaderElection13 PRE_VOTE round 0: submit vote requests at term 0 for -1: peers:[om1|ai205755.ams1o.com:9872, om3|ai205753.ams1o.com:9872, om2|ai205754.ams1o.com:9872]|listeners:[], old=null
[om1@group-5AEFCA43398D-LeaderElection13] INFO impl.LeaderElection: om1@group-5AEFCA43398D-LeaderElection13 got exception when requesting votes: java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[om1@group-5AEFCA43398D-LeaderElection13] INFO impl.LeaderElection: om1@group-5AEFCA43398D-LeaderElection13 got exception when requesting votes: java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[om1@group-5AEFCA43398D-LeaderElection13] INFO impl.LeaderElection: om1@group-5AEFCA43398D-LeaderElection13: PRE_VOTE REJECTED received 0 response(s) and 2 exception(s):
[om1@group-5AEFCA43398D-LeaderElection13] INFO impl.LeaderElection:   Exception 0: java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[om1@group-5AEFCA43398D-LeaderElection13] INFO impl.LeaderElection:   Exception 1: java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
[om1@group-5AEFCA43398D-LeaderElection13] INFO impl.LeaderElection: om1@group-5AEFCA43398D-LeaderElection13 PRE_VOTE round 0: result REJECTED
[om1@group-5AEFCA43398D-LeaderElection13] INFO server.RaftServer$Division: om1@group-5AEFCA43398D: changes role from CANDIDATE to FOLLOWER at term 0 for REJECTED
[om1@group-5AEFCA43398D-LeaderElection13] INFO impl.RoleInfo: om1: shutdown om1@group-5AEFCA43398D-LeaderElection13
[om1@group-5AEFCA43398D-LeaderElection13] INFO impl.RoleInfo: om1: start om1@group-5AEFCA43398D-FollowerState

[DaveTeng0]

hey @zorel , try clean up the folders defined in this config "ozone.om.ratis.storage.dir" (https://github.com/apache/ozone/blob/master/hadoop-hdds/common/src/main/resources/ozone-default.xml#L2039) on three om hosts,
usually it's like '/var/lib/hadoop-ozone/om/ratis'

[zorel]

Do I have to start 3 hosts? I try only with 1 scm and 1 om configured (but still the configuration for the cluster), and it fails anyway.

So I delete /data/ozone/om directory entirely, do the --init and ozone om.

And I still have the same issue.

[DaveTeng0]

hey @zorel , you probably need to check all om's (three om) log. Do they have any other exceptions other than that 'org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception'?

[DaveTeng0]

you might consider to enable debug logging as well, this will generate more verbose logs

[zorel]

The other OM are not even started. I've set the root logger to DEBUG and set the --debug flag on command line but it does not seems to produce more log.

[zorel]

Can this be of interest? First start after the --init:

2024-03-25 21:44:55,055 [om1-impl-thread1] INFO storage.RaftStorageDirectory: The storage directory /data/ozone/om/ratis/ba135e4e-69f0-3ec0-bcaf-5aefca43398d does not exist. Creating ...
2024-03-25 21:44:55,058 [om1-impl-thread1] INFO storage.RaftStorageDirectory: Lock on /data/ozone/om/ratis/ba135e4e-69f0-3ec0-bcaf-5aefca43398d/in_use.lock acquired by nodename [email protected]
2024-03-25 21:44:55,062 [om1-impl-thread1] INFO storage.RaftStorage: Storage directory /data/ozone/om/ratis/ba135e4e-69f0-3ec0-bcaf-5aefca43398d has been successfully formatted.
2024-03-25 21:44:55,062 [om1-impl-thread1] WARN util.FileUtils: Failed to Files.newInputStream /data/ozone/om/ratis/ba135e4e-69f0-3ec0-bcaf-5aefca43398d/current/raft-meta.conf with options []: java.nio.file.NoSuchFileException: /data/ozone/om/ratis/ba135e4e-69f0-3ec0-bcaf-5aefca43398d/current/raft-meta.conf

[zorel]

So, instead of doing a --init on one node and --bootstrap on the other, I did, once SCM is started, an ozone om --init on all the 3 om nodes, and then started them.
They eventually elected a leader in seconds.

[errose28]

it's the first run after the om --init

For the first time starting up, you will want to init all 3 OMs, not bootstrap. Bootstrapping is to go from an existing single node OM (who is already the leader of a one node Ratis quorum) to 3 OM nodes with Ratis.