Joe Lin

Jupyter Notebook on Google Colab

Google Colab is an excellent resource for running jupyter notebooks. It provides 4 CPUs, 16GB memory and an optional Nvidia Tesla K80 GPU for its users of up to 12-hour run time for free. Since there are plenty of tutorials on how to run your notebooks on Colab, I will only document some useful tips on working with Colab that makes it even more appealing.

Enable GPU on Colab

When training machine learning models, of course you want to have GPU(s). GPU can be enabled in Edit > Notebook Settings. Next, you want to make sure GPU is enabled.

# List all CPU and GPUs
from tensorflow.python.client import device_lib
device_lib.list_local_devices()

[Output]
[name: "/device:CPU:0"
 device_type: "CPU"
 memory_limit: 268435456
 locality {
 }
 incarnation: 17407044275154012122, name: "/device:GPU:0"
 device_type: "GPU"
 memory_limit: 11287966516
 locality {
   bus_id: 1
   links {
   }
 }
 incarnation: 13515505613363191723
 physical_device_desc: "device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7"]
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

You can see that a Tesla K80 (12GB memory) is available. You can also use this to verify tensorflow is using the GPU.

import tensorflow as tf
tf.test.gpu_device_name()

[Output]
'/device:GPU:0'
1
2
3
4
5

Use Google Drive as Data Storage

Every time you open a notebook, Google starts a new docker container, so everything will be lost once the notebook tab is closed. As a result, you want to retain your training results. The good news is you can mount your Google drive as an external disk onto Colab container! Even better, you can upload dataset to Google Drive and use it directly from Colab after mounting it.

To mount Google Drive, just execute following script in Colab notebook:

(This section will ask for access permission to Google Drive.)

# Setup authentication to mount Google Drive
!apt-get install -y -qq software-properties-common python-software-properties module-init-tools
!add-apt-repository -y ppa:alessandro-strada/ppa 2>&1 > /dev/null
!apt-get update -qq 2>&1 > /dev/null
!apt-get -y install -qq google-drive-ocamlfuse fuse
from google.colab import auth
auth.authenticate_user()
from oauth2client.client import GoogleCredentials
creds = GoogleCredentials.get_application_default()
import getpass
!google-drive-ocamlfuse -headless -id={creds.client_id} -secret={creds.client_secret} < /dev/null 2>&1 | grep URL
vcode = getpass.getpass()
!echo {vcode} | google-drive-ocamlfuse -headless -id={creds.client_id} -secret={creds.client_secret}
1
2
3
4
5
6
7
8
9
10
11
12
13

After gaining permission, you can mount Google Drive with following script:

# Execute when running on colab
# Mount Google Drive
!mkdir -p drive
!google-drive-ocamlfuse drive

import os
os.chdir('drive/Colab Notebooks')
1
2
3
4
5
6
7

Now your working directory is the Colab Notebooks directory in your Google Drive.

Upload / Download Files

If you want to upload / download files via browser, you can use Colab's python API.

from google.colab import files
files.download('file_path')
files.upload() # will display an upload button
1
2
3

← Jupyter Notebook Tips →