Ozone 1.4.0 will not connect to OM on Kubernetes: Failure in resolving OM host address

[RonaldHolex]

Hi,

When using Apache Ozone 1.4.0 or 1.3.0.7.2.18.0-273 from CDP (which I think is patched with HDDS-80411) then the cluster will start using a single SCM and OM. However, clients are unable to connect which was kind of confusing because doing a curl call on the OM port will return an HTTP response.

It turns out that a PR 1 introduced the usage of isReachable() [2] to test for connectivity in order to validate the settings.

However, it appears that this call is generally implemented using ICMP 3 (which confirms what I'm seeing with tcpdump) [4] when starting Impala 4.3.0.

Is this a known issue or a configuration issue?

I think ICMP is never going to work in Kubernetes and might even fail in some strictly firewalled environments. The check could be improved by opening a TCP connection to the proper port.

Kind regards,

[2]: if (!omHostAddress.getAddress().isReachable(5000)) {

A typical implementation will use ICMP ECHO REQUESTs if the privilege can be obtained, otherwise it will try to establish a TCP connection on port 7 (Echo) of the destination host.

[4]:
E0501 08:55:15.000583 1 OmUtils.java:851] Failure in resolving OM host address
Java exception follows:

java.io.IOException: OM host apache-ozone-om.apache-ozone unreachable in the URI
        at org.apache.hadoop.ozone.OmUtils.resolveOmHost(OmUtils.java:847)
        at org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:92)
        at org.apache.hadoop.fs.ozone.BasicRootedOzoneClientAdapterImpl.<init>(BasicRootedOzoneClientAdapterImpl.java:196)
        at org.apache.hadoop.fs.ozone.RootedOzoneClientAdapterImpl.<init>(RootedOzoneClientAdapterImpl.java:51)
        at org.apache.hadoop.fs.ozone.RootedOzoneFileSystem.createAdapter(RootedOzoneFileSystem.java:94)
        at org.apache.hadoop.fs.ozone.BasicRootedOzoneFileSystem.initialize(BasicRootedOzoneFileSystem.java:184)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3452)
        at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:162)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3557)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3504)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:522)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:270)
        at org.apache.impala.service.JniFrontend.checkFileSystem(JniFrontend.java:926)
        at org.apache.impala.service.JniFrontend.checkConfiguration(JniFrontend.java:832)
W0501 08:55:15.001075     1 FileSystem.java:3456] Failed to initialize fileystem ofs://apache-ozone-om.apache-ozone:9862/apache-impala/apache-impala/: java.io.IOException: OM host apache-ozone-om.apache-ozone unreachable in the URI

[kerneltime]

Thanks @RonaldHolex for bringing this up. cc @sadanand48

[sadanand48]

Thanks @RonaldHolex for the detailed description . HDDS-8041 was done in order to prevent endless retries when an invalid hostname is provided and the intention was to check connection to a host which is why ICMP looked like a good choice.
If ICMP is blocked, like you rightly pointed out we can check connectivity to the OM port by opening a TCP connection. However this is equivalent to setting retries=0 as even after the check , client will eventually establish a tcp connection (grpc).
I think we can introduce a config to use tcp that will be helpful in such cases or a config to even perform the host connection check would help in this case.
Is there a jira ticket or I can open one for this issue?

[RonaldHolex]

Hi Sadanand. There is no Jira ticket at this moment.

If the default for the config option you mentioned defaults to the default ICMP behaviour I would ask you to consider improving the error indicating that the ICMP ping failed.

I started tcpdump as a last resort after trying a lot of other options because it's generally not apparent that ICMP being used causes an cannot connect to host error (this error appeared to not be entirely true...).