ddp traing multi gpu Expected all tensors to be on the same device, but found at least two devices

[yxk9810]

File "/opt/conda/lib/python3.10/site-packages/torch/nn/functional.py", line 2199, in embedding
return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0! (when checking argument for argument index in method wrapper__index_select)

pytorch==1.12.0
transformers==4.20.0

runing in T4 x2gpu on kaggle: https://www.kaggle.com/code/jackiewu/notebook738aa0f5d2

[yxk9810]

runing command as:
! CUDA_VISIBLE_DEVICES=0,1 python -m tevatron.driver.train
--output_dir model_msmarco
--model_name_or_path bert-base-uncased
--save_steps 1000
--train_dir /kaggle/working/train_tevatron_100.json
--fp16
--per_device_train_batch_size 2
--train_n_passages 8
--learning_rate 5e-6
--q_max_len 64
--p_max_len 460
--num_train_epochs 3
--logging_steps 500
--overwrite_output_dir

[MXueguang]

will adding `--negatives_x_device ' helps?