🤔[question]tensorflow runtime error

[YUjunuuuuu]

Describe your question

I create a experiment about imagenet and the framework is tensorflow. However, it seems that there is some error in my experiment. The log file is below. We implements the keras.TFKerasTrial in model_def.py. However, it only trains for a while, then the error occurs. Can you give me some suggestions?

experiment_89_trial_83_logs.txt

[rb-determined-ai]

Sorry, I meant to answer you last week but got distracted.

Process 10 exit with status code 247

247 would indicate a process died due to being kill -9'd. If you didn't kill it, it likely was killed by the OOM killer. You might need to try a smaller batch size or something, or perhaps turn on profiling and watch your memory usage.

[YUjunuuuuu]

ok, I will try it and answer you later

[YUjunuuuuu]

I do experiments on imagenet and decrease the batch size to 128(gpu 3090*16, 8 batch per gpu), howerver, it dosen't work.Here are my config files.
[experiment_99_trial_93_logs.txt](https://github.com/determined-ai/det
exp99_distributed_yaml.txt