-
Notifications
You must be signed in to change notification settings - Fork 17
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
the network output will become nan #13
Comments
Hello, Normally, for each image, the expected output should be a binary map with just pixels equal to 0 or 1 but I guess that you have some masks that are not satisfying this constraint. This may explain your error ;) |
I am not sure why it is becoming nan suddenly. Can you share your training code ? I will check the IMTFE meanwhile. Maybe you have to force the output pixels to be in [0,1] in your loss using a clipping. Can you also check that all your target masks contain only 0 or 1 pixels ? |
I have some comments about your code that may explain your error : 1 - You need to provide probabilities to BCE_loss in "seg_out". To do so, you have to use "nn.Sigmoid()" and apply it to seg_out before passing it in the BCE_loss (Cf https://pytorch.org/docs/stable/generated/torch.nn.BCELoss.html) |
Hello, Did you fix your issues finally ? |
hello, i have same problem. i use nn.BCEWithLogitsLoss and my target masks contain only 0 or 1 pixels... Could you help me please? |
I think this could happen because some probabilities returned are too close to zero making the loss diverging. People usually add a small epsilon value to the prediction to prevent this divergence. Could you try that and tell me if it's fixed ? :) |
I am very grateful to the author for the pytorch version code. During my training, I found that the network output will become nan. Has the author encountered such a problem?
The text was updated successfully, but these errors were encountered: