get_pytorch_waymo_dataset.md

How to generate pytorch_block_nerf_dataset

1. download the Waymo Block dataset via the following command:

   pip install gdown # download google drive download.
   cd data
   gdown --id 1iRqO4-GMqZAYFNvHLlBfjTcXY-l3qMN5 --no-cache 
   unzip v1.0.zip
   cd ../

   The Google cloud may limit the download speed in this operation. You can instead: (1) Downloading in your browser by clicking this link. (2) Alternatively, you can directly download from the official Waymo website. However, this download may needs the sudo access to install the gsutil tool (if you don't have sudo access, you can download from your local laptop and then transport it to your server). The reference script is as follows:

   # install gsutil tool
   sudo apt-get install apt-transport-https ca-certificates gnupg # needs sudo access
   echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
   curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -
   sudo apt-get update && sudo apt-get install google-cloud-cli # needs sudo access
   gcloud init # login your google account then
   cd data
   gsutil -m cp -r \
     "gs://waymo-block-nerf/v1.0" \
     .
   unzip v1.0.zip
   cd ..

   You may otherwise symbol link the downloaded dataset ("v1.0") under the "data" folder. The Waymo official files (e.g., v1.0/v1.0_waymo_block_nerf_mission_bay_train.tfrecord-00000-of-01063) would be put under the data folder.

2. Transfer the original data in TF to the pytorch format via the following command:

   python data_preprocess/fetch_data_from_tf_record.py

3. Split the waymo dataset into blocks and extract corresponding information.

   python data_preprocess/split_block.py

Now you have finished the waymo data preprocess procedure and you can start training.