Advice on proper way to set ZMQ_MAX_SOCKETS

[arv1ndn]

We have a use-case where we routinely run 1000+ pods off a single enterprise-gateway instance.

In initial testing we hit a hard limit of around 200 pods. We tracked it down to default ZMQ_MAX_SOCKETS = 1024. We raised this limit higher by adding a couple lines to the RemoteMappingKernelManager as follows:

class RemoteMappingKernelManager(AsyncMappingKernelManager):
    """
    Extends the AsyncMappingKernelManager with support for managing remote kernels via the process-proxy.
    """
    def __init__(self, *args, **kwargs):
        super(RemoteMappingKernelManager, self).__init__(*args, **kwargs)
        self.shared_context = True  # always using a shared context
        self.context.set(zmq.MAX_SOCKETS, 1000000)

Ideally this parameter would be exposed in a standardized manner for users to set/change. Does such a mechanism exist already?

[kevin-bates]

Hi @arv1ndn - this is really interesting and I'm really glad you were able to get past your threshold of 200 pods!

The ZMQ context is not configurable itself but I think we have a couple of approaches.

The most simple would be to extend RemoteMappingKernelManager similar to what you have done but rather by adding a validator associated with the context instance.

The other would be to add support for configuring MAX_SOCKETS in jupyter_client, but because it's really a function of the context instance (that should be a shared instance, which is configurable), I'm not entirely sure how straightforward that would be.

One of the advantages of having our own MultiKernelManager subclass, and because this kind of thing really only applies to EG (for now), is that we can do this kind of thing without disturbing the default framework and use EG as a kind of "test ground".

Since shared_context is configurable, and its default value is True, you could remove that setting, or ensure your EG instance is started with --RemoteMappingKernelManager.shared_context = True.

I've gone ahead and coded up what the context validator would look like below. By making both of these changes, you should be able to remove the initializer completely.

    # Hook the validation of the ZMQ context object to set MAX_SOCKETS is the connection is shared
    @validate("context")
    def _context_validate(self, proposal: Dict[str, str]) -> Context:
        zmq_context = proposal["value"]
        try:
            # First, assert we have a ZMQ context instance
            assert isinstance(zmq_context, Context)
        except ValueError:
            raise TraitError(f"Invalid ZMQ Context instance! '{type(zmq_context)}'")

        if self.shared_context:
            zmq_context.set(MAX_SOCKETS, int(os.getenv("EG_MAX_ZMQ_SOCKETS", 1024)))
            self.log.debug(f"Set ZMQ sockets to {zmq_context.MAX_SOCKETS}")
        return zmq_context

Please note that I have not exercised this validator so please take that into account, although this should work. You can also see that I kept the 1024 as a default value, allowing overrides via the env EG_MAX_ZMQ_SOCKETS - which should be easy to "flow" via the helm charts (or otherwise).

If you find this successful, please open a pull request so we can include it in our upcoming 3.0 release, which should be entering its RC phase shortly (hopefully next week).

Please feel free to continue the discussion should you run into issues or need more information.

[kevin-bates]

Hmm, the type-hint may want to change to Dict[str, Context], although it probably won't matter much, but more correct.

[kevin-bates]

Also, we should change the default in the os.getenv() call to 1024 (which, I've edited in the previous response - sorry about that.). If you know of a constant associated with MAX_SOCKETS, we should use that if possible.

[arv1ndn]

Ok I'll try to raise a PR on this today. Unfortunately while libzmq does define ZMQ_MAX_SOCKETS_DLFT, I was unable to find a python binding for this value in pyzmq. I think we might need to end up replicating the value (1023).

[arv1ndn]

Hi @kevin-bates, thank you for all the info above. Does this look reasonable to you? #1154

I'll respond to any comments directly in the PR.