-
Notifications
You must be signed in to change notification settings - Fork 10
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Training automatically closes #10
Comments
by the way, i use pascal voc2007 training dataset. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
hello,author.
i meet a question when i run your code(GTX 1080 Ti), sometime report this error
2019-12-12 14:41:58.024277: E tensorflow/stream_executor/cuda/cuda_dnn.cc:81]
2019-12-12 14:41:58.024505: I tensorflow/stream_executor/stream.cc:4818] stream 0x560dcbf2bd50 did not memzero GPU location; source: 0x7f2a617f8d40
2019-12-12 14:41:58.024537: I tensorflow/stream_executor/stream.cc:4818] stream 0x560dcbf2bd50 did not memzero GPU location; source: 0x7f2a3bffdd40
2019-12-12 14:41:58.024574: I tensorflow/stream_executor/stream.cc:4818] stream 0x560dcbf2bd50 did not memzero GPU location; source: 0x7f2a3bffdd40
2019-12-12 14:41:58.024542: I tensorflow/stream_executor/stream.cc:4818] stream 0x560dcbf2bd50 did not memzero GPU location; source: 0x7f2a617f8d40
2019-12-12 14:41:58.027290: I tensorflow/stream_executor/stream.cc:1936] stream 0x560dcbf2bd50 did not enqueue 'start timer': 0x7f2a61ff9d30
2019-12-12 14:41:58.027373: I tensorflow/stream_executor/stream.cc:1947] stream 0x560dcbf2bd50 did not enqueue 'stop timer': 0x7f2a61ff9d30
2019-12-12 14:41:58.027400: I tensorflow/stream_executor/stream.cc:4818] stream 0x560dcbf2bd50 did not memzero GPU location; source: 0x7f2a61ff9d40
2019-12-12 14:41:58.027426: I tensorflow/stream_executor/stream.cc:4818] stream 0x560dcbf2bd50 did not memzero GPU location; source: 0x7f2a61ff9d40
2019-12-12 14:41:58.027456: I tensorflow/stream_executor/stream.cc:4818] stream 0x560dcbf2bd50 did not memzero GPU location; source: 0x7f2a3b7fcd40
2019-12-12 14:41:58.027499: I tensorflow/stream_executor/stream.cc:4818] stream 0x560dcbf2bd50 did not memzero GPU location; source: 0x7f2a617f8d40
2019-12-12 14:41:58.027884: I tensorflow/stream_executor/stream.cc:4818] stream 0x560dcbf2bd50 did not memzero GPU location; source: 0x7f2a61ff9d40
2019-12-12 14:41:58.027907: I tensorflow/stream_executor/stream.cc:4818] stream 0x560dcbf2bd50 did not memzero GPU location; source: 0x7f2a61ff9d40
2019-12-12 14:41:58.027926: I tensorflow/stream_executor/stream.cc:4818] stream 0x560dcbf2bd50 did not memzero GPU location; source: 0x7f2a61ff9d40
2019-12-12 14:41:58.027944: I tensorflow/stream_executor/stream.cc:4818] stream 0x560dcbf2bd50 did not memzero GPU location; source: 0x7f2a61ff9d40
2019-12-12 14:41:58.027962: I tensorflow/stream_executor/stream.cc:4818] stream 0x560dcbf2bd50 did not memzero GPU location; source: 0x7f2a61ff9d40
2019-12-12 14:41:58.028478: I tensorflow/stream_executor/stream.cc:4818] stream 0x560dcbf2bd50 did not memzero GPU location; source: 0x7f2a61ff9d40
2019-12-12 14:41:58.028501: I tensorflow/stream_executor/stream.cc:4818] stream 0x560dcbf2bd50 did not memzero GPU location; source: 0x7f2a61ff9d40
2019-12-12 14:41:58.028519: I tensorflow/stream_executor/stream.cc:4818] stream 0x560dcbf2bd50 did not memzero GPU location; source: 0x7f2a61ff9d40
2019-12-12 14:41:58.028537: I tensorflow/stream_executor/stream.cc:4818] stream 0x560dcbf2bd50 did not memzero GPU location; source: 0x7f2a61ff9d40
2019-12-12 14:41:58.028555: I tensorflow/stream_executor/stream.cc:4818] stream 0x560dcbf2bd50 did not memzero GPU location; source: 0x7f2a61ff9d40
2019-12-12 14:41:58.028829: I tensorflow/stream_executor/stream.cc:4818] stream 0x560dcbf2bd50 did not memzero GPU location; source: 0x7f2a61ff9d40
2019-12-12 14:41:58.028898: I tensorflow/stream_executor/stream.cc:4818] stream 0x560dcbf2bd50 did not memzero GPU location; source: 0x7f2a61ff9d40
2019-12-12 14:41:58.029039: I tensorflow/stream_executor/stream.cc:4818] stream 0x560dcbf2bd50 did not memzero GPU location; source: 0x7f2a617f8d40
2019-12-12 14:41:58.029051: I tensorflow/stream_executor/stream.cc:302] did not allocate timer: 0x7f2a617f8d30
2019-12-12 14:41:58.029058: I tensorflow/stream_executor/stream.cc:1936] stream 0x560dcbf2bd50 did not enqueue 'start timer': 0x7f2a617f8d30
2019-12-12 14:41:58.029069: I tensorflow/stream_executor/stream.cc:1947] stream 0x560dcbf2bd50 did not enqueue 'stop timer': 0x7f2a617f8d30
2019-12-12 14:41:58.029076: F tensorflow/stream_executor/cuda/cuda_timer.cc:65] Check failed: start_event_ != nullptr && stop_event_ != nullptr
Aborted (core dumped)
sometimes reports this error:
InternalError (see above for traceback): cuDNN Backward Filter function launch failure : input shape([1,256,150,200]) filter shape([3,3,256,512])
[[Node: tower_0/gradients/tower_0/build_rpn/rpn_conv/3x3/Conv2D_grad/Conv2DBackpropFilter = Conv2DBackpropFilter[T=DT_FLOAT, _class=["loc:@tower_0/gradients/AddN_20"], data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](tower_0/build_pyramid/fuse_P2/BiasAdd, ConstantFolding/tower_0/gradients/tower_0/build_rpn/rpn_conv/3x3/Conv2D_grad/ShapeN-matshapes-1, tower_0/gradients/tower_0/build_rpn/rpn_conv/3x3/Relu_grad/ReluGrad, ^tower_0/gradients/tower_0/build_rpn/rpn_conv/3x3/BiasAdd_grad/BiasAddGrad)]]
I can't find the answer from Google,could you help me? thank you very much.
The text was updated successfully, but these errors were encountered: