hello，i have a question！

[xiaolongxia520]

I used your training parameters, loss=nan?I checked and said to use dynamic learning rate. Have you used it?

[kritikaksu]

Hey..The same problem was with me then i realized that my input image size was not 112x112. when i have re-scaled them to 112x112 then Nan Values have been gone..

[GranMin]

hello,My image size is 96x96, and I changed the input of the model, just as shown blow. But my loss is still nan. Do you have any idea about it?

[Androsimus]

@GranMin
Hi!
In a dataset.py size of input image is hardcoded in function _transform_images. Maybe this is the reason of nans.

[Androsimus]

@GranMin besides, if you use script convert_train_binary_tfrecord.py for data preprocessing, you must enforce all dir names of your N classes are strictly within interval [0,...,N-1], because dir names will be further used as labels in Softmax loss.

[GranMin]

@Androsimus thankyou, I find my problem that I labeled my dataset by [1,...,N] rather than [0,...,N-1]. So I got nan. I correct it and then it runs well.
By the way, when I changed optimizer to Adam, model resnet50 sometimes achieve accuracy of 0.9, but sometimes sticks in 0.3, do you have any idea about it?

[Androsimus]

@GranMin if you use identical hyperparameters and datasets, then I can hardly imagine appropriate reasons.
But if you use different learning rates for example or something else, then I can try to explain this.

[GranMin]

@Androsimus thanks.I had some problem with the understanding of tf.optimizers, I changed the hyperparameters as somebody said, and it works. Best wishes!