gdcp (Google Drive cp) is a command-line tool that supports robust and fast file transfers to and from Google Drive, somewhat analogous to using scp to transfer files to and from a Linux/Unix server. It is designed to be used in a workflow where large data sets stored in Google Drive are downloaded to a server for processing and results are uploaded back to Google Drive.
Notable features:
- supports large file transfers
- MD5 checksum verification
- recursive folder upload and download
- recursive depth-limited folder listing
- recursive ownership transfer
- compensates for errors during uploads and downloads, resuming partial transfers when possible and retrying complete transfers otherwise
gdcp is a python script which depends on
- Python 2.7. Python 3 is not supported. If you see syntax errors about exception handling or print statements make sure
gdcpis not being run with Python 3. - PyDrive
- backoff
The easiest way to install Python dependencis is with pip.
$ pip install pydrive
$ pip install backoff
OS X Note PyDrive and its dependencies (notably six) may not install correctly in OS X when using system Python and pip. To get around this, use a non-system Python (e.g. from MacPorts or Homebrew) and/or use virtualenv. See pypia/pip#2468 for more details.
If pip is not already installed on your system, it should be possible to install it with easy_install.
$ easy_install pip
These commands may require superuser privileges (e.g. sudo) if you wish to install to system locations. Otherwise, install libraries with --user to install to your home directory or install libraries in an isolated Python environment with virtualenv.
Now place gdcp somwhere in your path. For example, if ~/bin is in your path
$ git clone https://github.com/ctberthiaume/gdcp.git
$ cp gdcp/gdcp ~/bin
The first time you run gdcp a new directory ~/.gdcp/ will be created and you'll be greeted with the following instructions for obtaining your own OAuth2 client ID.
- Visit https://console.developers.google.com/
- Create a new project and select it
- Under 'APIs' make sure the 'Drive API' is turned on
- Under 'Credentials' create a new OAuth client ID
Choose 'Installed -> Other' for application type
- Click 'Download JSON' to download the secrets file
- Copy the secrets file to ~/.gdcp/client_secrets.json
Once you've created ~/.gdcp/client_secrets.json, run gdcp again to authorize access to your Google Drive. You should see something like
Go to the following link in your browser:
followed by a long URL. Visiting this URL in a browser where you're already logged into your Google account will yield a verification code. Copy and paste this code into the terminal to complete the authentication process. A new file, ~/.gdcp/credentials.json, will be created which grants access to your Google Drive files. If you want to use gdcp on a different computer without going through authentication again, just copy ~/.gdcp/ to the new computer.
List file metadata. For example, to list information about a Googel Drive folder with ID 0Bxt5Ia3JxzdHfkJDeUxCQ3RyaWp.
$ gdcp list -i 0Bxt5Ia3JxzdHfkJDeUxCQ3RyaWp
foo 0Bxt5Ia3JxzdHfkJDeUxCQ3RyaWp folder
foo/bar.txt 0Bxt5Ia3JxzdHZGtabW9xTEduSkE file 7 14758f1afd44c09b7992073ccf00b43d
This prints information about that folder and its first-level contents. Tab-delimited columns are title, id, type, fileSize, md5Checksum. Type will be file or folder for downloadable files and folders. For files created with a Google Apps app - documents, spreadsheets, maps, third-party apps, etc - the type reported will be the name parsed from the file's MIME type. For example, a Google Sheets file with MIME type application/vnd.google-apps.spreadsheet has type spreadsheet. To increase the depth of the recursive metadata listing specify a --depth value greater than 0.
Recursively upload files or folders. For example, to upload a local folder into a Google Drive parent folder with ID 0Bxt5Ia3JxzdHfkJDeUxCQ3RyaWp.
$ gdcp upload -p 0Bxt5Ia3JxzdHfkJDeUxCQ3RyaWp ./subfolder
subfolder/
subfolder/subfile.txt
100.00% 209715200 44.08MB/s 4.76s MD5...OK
Uploaded 2 file(s) and folder(s)
By default gdcp will compare the MD5 checksum of local files against the checksum reported by Google after upload.
Recursively download files or folders. For example, to download a Google Drive folder with ID 0Bxt5Ia3JxzdHfkJDeUxCQ3RyaWp.
$ gdcp download -i 0Bxt5Ia3JxzdHfkJDeUxCQ3RyaWp .
./foo/subfolder/
./foo/subfolder/subfile.txt
100.00% 209715200 50.32MB/s 4.17s MD5...OK
./foo/bar.txt
100.00% 7 0.00MB/s 0.49s MD5...OK
Downloaded 4 file(s) and folder(s)
By default gdcp will compare the MD5 checksum of local files against the checksum reported by Google after download.
Recursively transfer ownership of files or folders to another Google Apps account. For example, to transfer folder 0Bxt5Ia3JxzdHfkJDeUxCQ3RyaWp and all of its contents.
gdcp transfer -i 0Bxt5Ia3JxzdHfkJDeUxCQ3RyaWp -e [email protected]
foo 0Bxt5Ia3JxzdHfkJDeUxCQ3RyaWp
subfolder 0Bxt5Ia3JxzdHcVJxVTRfYVdOdmc
subfile.txt 0Bxt5Ia3JxzdHN0NqUEh3RnJic28
bar.txt 0Bxt5Ia3JxzdHZGtabW9xTEduSkE
Transferred ownership for 4 file(s)