[Load token classification model from checkpoint get No such file or directory: 'megatron-bert-cased_encoder_config.json']

[bugface]

Describe your question

After training token classification model, several checkpoints .ckpt were generated along with .nemo. Load .nemo is ok. But I tried to load checkpoints via:

nemo_nlp.models.TokenClassificationModel.load_from_checkpoint(my_chk_path)

but give an Error:

No such file or directory: 'megatron-bert-cased_encoder_config.json'

I unpacked the .nemo and found 5 files:

label_ids.csv
megatron-bert-cased_encoder_config.json
model_config.yaml
model_weights.ckpt
vocab.txt

I did not know where did the megatron-bert-cased_encoder_config.json come from?

Environment overview (please complete the following information)

using nemo docker from ngc and at BRANCH = 'r1.0.0rc1'

[ericharper]

Hi, restoring NLP models can be complicated as there are potentially many different artifacts needed for restore. This is why we've created the .nemo file and .restore_from. We mainly use the PyTorch Lightning Checkpoints to auto-resume from training. They can be used in other workflows outside of NeMo, but then the user would need to add custom implementation for restore.

And to answer your question "I did not know where did the megatron-bert-cased_encoder_config.json come from?"
Megatron models need certain artifacts to be instantiated, mainly the config and vocab. We generate both of these and automatically save them in the .nemo file when using Megatron models in NeMo NLP.

Also, we will be updating the NeMo NLP documentation with this information in the near future. Sorry for the current inconvenience.

[bugface]

The current implementation use val_loss to select the best checkpoint to save during the training. However, especially for NER, many previous works used F1 score which could be a better metric for checkpoint selection. I did not see any options for F1 score or other metrics besides the val_loss in the documentation. I would like to know if such metrics are available?