Getting files from our own google drive with API

A recap and complete example on how this is done.

Updated link of scripts and permissions

Geek Level: extreme

Background story: I have 24TB of google drive storage, which some of the files I want to transfer to my dedicated servers in Netherlands and Singapore.

At first, the easiest way to download, is by changing the permission of files you want download as public (“anyone on the internet with this link can view”), and then in Linux server, we can use the utility called gdown.

https://pypi.org/project/gdown

How to use gdown is pretty much simple. You need to transform the google shared link into certain format, from

https://drive.google.com/file/d/abc123456789xxx/view?usp=sharing

to

https://drive.google.com/uc?id=abc123456789xxx

And then call it from gdown:

bash$ gdown 'https://drive.google.com/uc?id=abc123456789xxx'

To make it easy, we can create small script that can change format automatically:

cat gdownload.sh 
#!/bin/bash
param=$1;
fileid=`echo $param | grep -oP '(?<=file/d/).+(?=/view)'`;
echo $fileid;
gdown "$fileid"

DONE.

But… after a while, google will start suspecting that the server is actually a “robot” and forbid you to use gdown anymore to download any link. Either that, or you will get quota limitation issue.

So, there is no other way, we have to use proper API to download the files, and authenticated to Google.

I had been reading the https://developers.google.com/drive/api/v3/quickstart/python and had been trying the example, but I have to be honest, Google’s documentation sucks — deep.

After few hours without success, many trial errors, opening a lots of StackOverflow, finally I made it. Therefore I decided to share here, so that you don’t fall into same pitfall.

First phase, is to enable the Drive API. You can open the link above, and click the button “Enable the Drive API”. You will be asked for the project name, enter as you like, and for the type, you choose either “TV” or “Desktop”. At the end of creation, please download the “credentials.json” and save it.

The file is meant for initial authentication method for a “robot” client. And is not a key or token to call the API yet. We will use this “credentials.json” to create token later on.

As for the python, we will use V3, and first need to install pip module if not yet there:

sudo apt install python3-pip

Then we install the needed python libraries to access google drive API. Here I found that example from google was missing needed file. This is the complete one, we will run the pip as module instead as standalone program:

bash$ python3 -m pip install google-api-python-client google-auth-httplib2 google-auth-oauthlib testresources

OK, now we need the code, here’s the working example:

gd.py

How to use it:

python3 gd.py "fileid"

We can combine the script posted earlier in this article “gdownload.sh” with gd.py so that it will translate shared link into File ID and will call gd.py directly.

BTW you will need to place “credentials.json” at same location with “gd.py”. How the gd.py works:

1. check if “token.pickle” exists or not. If exists it will try to authenticate with that, and the drive API can be used directly.
2. If not exists, or invalid, it will try searching for “credentials.json”. The information inside will be used to do “human precheck”. Google does not allow the token to be created without human intervention.
3. The script either will open your default-browser, and you have to authenticate there, or will only show you the link — if you run from server cli (and you have to copy paste the link manually in browser). If you do the latter option (browser login from different machine), you have to copy paste again the produced link (localhost address) into your server by using curl while the gd.py is running.
4. If successful, a token.pickle will be produced and next time gd.py is running, it won’t need human intervention anymore. The steps inside gd.py already ensures that token gets refreshed if expired.
5. Process file ID download and show the progress percentage.

That’s all my friends. This example can be used as starting point to utilize other drive API capabilities as needed.

Thanks a lot for reading 👍