请问一下,我训练的时候中途会终止,报下面的错误,应该怎么改?

[wenliangsun]

out of memory
invalid argument
an illegal memory access was encountered
an illegal memory access was encountered
2018-12-19 21:32:23.621928: E tensorflow/stream_executor/cuda/cuda_event.cc:49] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS
2018-12-19 21:32:23.621968: F tensorflow/core/common_runtime/gpu/gpu_event_mgr.cc:203] Unexpected Event status: 1
2018-12-19 21:32:23.621983: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:639] failed to record completion event; therefore, failed to create inter-stream dependency
已放弃 (核心已转储)

[yangxue0827]

Something wrong with r-nms, I haven't found the reason yet. @wenliangsun

[zuowanghanguang]

@wenliangsun Hello,I meet the same issue as you ,do you know how to solve this problem?Thank you!

[wenliangsun]

Hi，I have not found a solution at present. @zuowanghanguang

[Jnyle]

did you find the solution yet?

[Icecream98]

did you find the solution yet?

I have the same issue, did you solve it?

[cpaaax]

Something wrong with r-nms, I haven't found the reason yet. @wenliangsun

Hi,have you found the solutions?

[FishBigOcean]

Does anyone know how to solve this problem?