Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How can I connect previous log file into new one in tensorboard #679

Open
mohanades opened this issue Dec 31, 2022 · 1 comment
Open

How can I connect previous log file into new one in tensorboard #679

mohanades opened this issue Dec 31, 2022 · 1 comment

Comments

@mohanades
Copy link

Hi,
I trained my model for 50 epochs. Now I want to continue training for other 15 epochs. so I start training with model.load_from_checkpoint("path")
but I don't know how to show continuation of logs with tensorboard
I thought if I write same path of previous file logs, it continue but it didn't show me the previous logs.

I'd be appreciate if you could help me because I need to see the previous training logs along with the new ones.
Part of source code is:

checkpoint_callback=ModelCheckpoint(dirpath="model",filename="newmodel_1",save_last=True,verbose=True,monitor="val_ce_loss",mode="min")
logger=TensorBoardLogger("training-logs",name="test_1")
model1 = model(config)
model_with_previous = model1.load_from_checkpoint(base_path_ckpt)
trainer = pl.Trainer(
     max_steps = max_steps,
     max_epochs=N_EPOCHS,
     gpus=(1 if torch.cuda.is_available() else 0),
     logger=logger,
     callbacks=[checkpoint_callback], 
)
%load_ext tensorboard
%tensorboard --logdir ./lightning_logs
@lanpa
Copy link
Owner

lanpa commented Jan 2, 2023

Hi, you might need to pass the purge_step parameter.
Here is an low level API example:
https://github.com/lanpa/tensorboardX/blob/master/examples/demo_purge.py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants