Subsequent hparams missing

[rijobro]

Describe the bug
Any hparams not present in the first run will never be shown. This is frustrating if extra information is added as you realise it might be relevant during the experiment. You may also have different experiments in the same folder.

Minimal runnable code to reproduce the behavior

# Example 1
from tensorboardX import SummaryWriter
with SummaryWriter() as w:
    w.add_hparams({'param1': 0.1}, {"metric1": 1})
    w.add_hparams({'param1': 2, "param2": 4}, {"metric1": 3, "metric2": 2}) 

rm -rf runs/*

# Example 2
from tensorboardX import SummaryWriter
with SummaryWriter() as w:
    w.add_hparams({'param1': 0.1}, {"metric1": 1})
with SummaryWriter() as w:
    w.add_hparams({'param1': 2, "param2": 4}, {"metric1": 3, "metric2": 2})

Expected behavior
The first example shows param1, param2, metric1, metric2. The second only shows param1 and metric1. I would expect the tables to contain all parameters and metrics present in all runs. They can be displayed as blanks if not present for a given run, as is the case for the first example.

Screenshots
Example 1

Example 2

Environment

protobuf                      3.19.4
pytorch-ignite                0.4.9
tensorboard                   2.10.0
tensorboard-data-server       0.6.1
tensorboard-plugin-wit        1.8.1
tensorboardX                  2.5.1
torch                         1.12.1
torchvision                   0.13.1

Python environment
Conda 3.9

[lanpa]

Thank you for the bug report. I have tried your example but find that I have an opposite result compare to yours. (Example 1 has four columns, Example 2 has two columns) Then I tried to restart the tensorboard several times without re-running your snippet. Surprisingly, the missing column appears from time to time. Finally, I added --load_fast=false to the tensorboard command and all the columns appears as expected (10 success out of 10 trial). Can you confirm this for me?

[rijobro]

Thanks for getting back to me. You're right that it appears to be random. Unfortunately, for me, even with load_fast=false, the behaviour still seems to be random. The extra columns appear in about 50% of cases. Interestingly, stopping and starting the tensorboard server whilst keeping the same directory doesn't result in any randomness. The randomness seems to appear when the directory is re-created (i.e., rm -rf runs).

[rijobro]

Hi @lanpa , any updates on this one?

[lanpa]

Sorry having missed your reply for so long. After seeing your reply I tried launching tensorboard for several times with the --load_fast=false arg, and I can get the randomness like yours. So --load_fast=false is not able to solve this. However the runs folder is never altered during my experiments, I would need some time to dig into this. p.s. I have also tried tensorboard v2.4.0, which has no load_fast option available. The extra columns never appear. Chances are tensorboard has changed the display logic.