Challenge connecting to jstatd running in k8s

[apple-corps]

With a bit of effort I'm able to connect visualVM to a java process running on kubernets, by forwarding the jmx port and using a configuration like below for starting my java process

-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=9010 -Dcom.sun.management.jmxremote.rmi.port=9010 -Djava.rmi.server.hostname=127.0.0.1

Then I port forward 9010 to my localhost via kubectl and add localhost:9010 as a JMX host in VVM and Magic , it works!

Underneath Local, my k8s PID appears with visual statistics!

However, I really wanted to use VVM to observe GC stats on the device. So I added the VGC plugin. But I get the Not supported for this JVM when

So I started jstatd on the server like

jstatd -p 1099 -J-Djava.security.policy=<(echo 'grant codebase "jrt:/jdk.jstatd" {permission java.security.AllPermission;};grant codebase "jrt:/jdk.internal.jvmstat" {permission java.security.AllPermission;};') -J-Djava.rmi.server.hostname=127.0.0.1 -J-Djava.rmi.server.logCalls=true

And then I tried forwarding that port similar to above. And then I try to right click Local and addJStatD from the menu. But suprise, it doesn't work.

So I went down a rabbit hole trying to figure out why it didn't work. And I read elsewhere it did not work because jstatd really opens up 3 ports when it starts. And supposedly there were hacks to resolve this.

So I took the latest hack and built a release for it (long story). https://github.com/apple-corps/alt-jstatsd-build-image . Now I could try to run this hack'd jstatd binary and set and forward all the ports hoping that VVM would connect. And did it work, it did not :(

For completeness sake, here's how I ran the hack

java -cp "./ejstatd.jar:$JAVA_HOME/lib/tools.jar" -Djava.rmi.server.hostname=127.0.0.1 -Djava.rmi.server.logCalls=true com.github.anthony_o.ejstatd.EJstatd -pr2222 -ph2223 -pv2224

And then through the logging I did see the following below in the jstatd log created by logCalls.

Dec 21, 2023 12:10:09 PM sun.rmi.server.UnicastServerRef dispatch
FINER: RMI TCP Connection(5)-127.0.0.1: [127.0.0.1: com.github.anthony_o.ejstatd.RemoteHostImpl[-463e121c:18c8c36aaf3:-7fff, 5020496887619864873]: public abstract int[] sun.jvmstat.monitor.remote.RemoteHost.activeVms() throws java.rmi.RemoteException,sun.jvmstat.monitor.MonitorException]
Dec 21, 2023 12:10:11 PM sun.rmi.transport.tcp.TCPTransport handleMessages
FINE: RMI TCP Connection(83)-127.0.0.1: [127.0.0.1] exception: 
java.rmi.NoSuchObjectException: no such object in table
	at java.rmi/sun.rmi.transport.Transport.serviceCall(Transport.java:177)
	at java.rmi/sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:562)
	at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:796)
	at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:677)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:676)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
nsport.java:677)
Dec 21, 2023 12:57:19 PM sun.rmi.server.UnicastServerRef dispatch
FINER: RMI TCP Connection(41)-127.0.0.1: [127.0.0.1: com.github.anthony_o.ejstatd.RemoteHostImpl[-56efd75a:18c8c6e2052:-7fff, 4917613067382858353]: public abstract int[] sun.jvmstat.monitor.remote.RemoteHost.activeVms() throws java.rmi.RemoteException,sun.jvmstat.monitor.MonitorException]
Dec 21, 2023 12:57:22 PM sun.rmi.server.UnicastServerRef dispatch
FINER: RMI TCP Connection(41)-127.0.0.1: [127.0.0.1: com.github.anthony_o.ejstatd.RemoteHostImpl[-56efd75a:18c8c6e2052:-7fff, 4917613067382858353]: public abstract int[] sun.jvmstat.monitor.remote.RemoteHost.activeVms() throws java.rmi.RemoteException,sun.jvmstat.monitor.MonitorException]

Regardless I guess I'm explaining my ignorance on why it doesn't work.

Describe the solution you'd like

If I'm able to connect to VVM via JMX as above, I'd similarly like to get the VGC stats.

[thurka]

The problem with jstatd opening additional random port for communication is already addresses in JDK. See https://bugs.openjdk.org/browse/JDK-8196729 . So with this fix, it should be possible to use jstatd in kubernetes.