Download a folder with more than 50 files

[GeorgeBatch]

I tried downloading a folder from Google Drive with more than 50 files. Is there a way to use gdown to download such a folder?

[GeorgeBatch]

I thought that if I could get the links/ids of all small folders (<50 files) and files, then I will be able to iterate through them and download the contents of the full folder. I also thought that recording the download status is a good idea in case the process interrupts so that you don't need to start from scratch.

Here is how you can get the links to files/folders: StackExchange Answer.

I recorded my attempt of using it in TCGA-lung-download repository. It did not fully work since after some time I started running into the problem described in issue#43 for gdown library.

Do you have any idea how I can catch the Access Denied ... warnings as Exceptions/Errors so that I can at least record that the folder/file was not successfully downloaded. This way, one would be able to resume the process after Google grants access again in 24 hours.

[wkentaro]

gdown.download returns None when it fails, whereas it returns file path if succeed. Does this help?

gdown/gdown/download.py

Line 183 in 12217e3

return

[GeorgeBatch]

Thank you! I will try to use this somehow. It does not solve the problem straight away since I was using gdown.download_folder() to download folders one by one from within this folder in Google Drive. I guess I could use the return from the gdown.download_folder() function, but I am not sure if the file name will still be added or not if the access is denied by Google.

gdown/gdown/download_folder.py

Line 359 in 12217e3

return filenames

By the way, why do you limit the number of downloadable files within one folder to 50?

gdown/gdown/download_folder.py

Line 27 in 12217e3

MAX_NUMBER_FILES = 50

[wkentaro]

This is why: #90 (comment)

[GeorgeBatch]

This is why: #90 (comment)

Makes sense! Thank you!

[tgandor]

Hm, this would be interesting. rclone can somehow do it, but I needed to enable API access for it (there were some hints about doing it anonymously; there was also something about this being slower than getting metatada from the API).

An alternative would be to script gdown for downloading multiple files; the URLs for a large folder could be gathered using something like this:
https://webapps.stackexchange.com/questions/88769/get-share-link-of-multiple-files-in-google-drive-to-put-in-spreadsheet (answer about using JS to store the URL list in a google spreadsheet).

[GeorgeBatch]

Hm, this would be interesting. rclone can somehow do it, but I needed to enable API access for it (there were some hints about doing it anonymously; there was also something about this being slower than getting metatada from the API).

An alternative would be to script gdown for downloading multiple files; the URLs for a large folder could be gathered using something like this: https://webapps.stackexchange.com/questions/88769/get-share-link-of-multiple-files-in-google-drive-to-put-in-spreadsheet (answer about using JS to store the URL list in a google spreadsheet).

Have not tried rclone, but it looks like it should work. The solution with writing the script did not work - looks like Google just starts blocking access to files from the same folder. See issue#43 in the gdown library.

[sassbalint]

It could be a good workaround if gdown FOLDER_URL --folder --list or something like this could provide a list of URLs for all of the files sitting in FOLDER_URL. After this, the files could be processed one by one.

[hunterbuchanan]

It seems like the 50 file limitation is due to the way gdown receives the file list for each folder from Google, which is similar to how the first 50 files show up in Google Drive in a browser window before you scroll down and the next bunch appear.

In other words, I think the problem is that gdown can't even get the list of files past 50 in the first place. If they could list all the files, they could download all the files. You probably need to do it the fancy API way to get past 50, which requires something like rclone or just using the Google API directly.

Edit: Having said that, I still would very much support gdown adding this capability if someone could figure out a solution. I love how much easier gdown is than rclone, but with a 50 file limit per folder, it doesn't work in all use cases.

[ryankert01]

Looks like there's probably a way that can recursively get all file url? anyone have idea?