Configuring and Using Rclone

Overview

Teaching: 10 min
Exercises: 10 min

Questions

How do I Configure Rclone?

Objectives

Make Rclone ready for data transfer with gdrive

Using Rclone

Rclone is a free utility for syncing directories between object storage systems (such as Amazon S3, Dropbox, Google Drive etc) and file based storage (e.g. home or scratch))

https://rclone.org

Rclone in MANA

Rclone is installed on the Mana Data Transfer Nodes and can be used in the command line via

$ rclone

Configuring Rclone

Before you can use Rclone, you must configure it This configuration step will set up access for the remote object storage system that you want to transfer data to and from In this tutorial we will configure Google Drive since UH has Google for Education and everyone at UH has it

Open a Shell Session on MANA

1.) Start a shell session on MANA through your own terminal or you can use Open OnDemand via https://mana.its.hawaii.edu

Clusters -> >_Mana_Shell_Access

From your terminal/shell ssh to one of the Mana DTNs

$ ssh username@hpc-dtn1.its.hawaii.edu

*You may be prompted for your password depending on where you are SSHing from and you WILL be prompted for DUO two-factor verification.

Two Factor Authentication

Example

$ ssh luketn@hpc-dtn1.its.hawaii.edu

Two Factor Authentication for Duo Push

$ 1

Two Factor Authentication for Phone Call

$ 2

Two Factor Authentication for SMS

$ 3

Configuring Rclone

$ rclone config

$ n

No remotes found - make a new one

n) New remote

s) Set configuration password

q) Quit config

Choose a name for the remote object storage system

You’ll be prompted for the name of the remote object storage system\, we use “rclone-gdrive” in this tutorial

$ rclone\-gdrive

Choosing a Storage Option

$ 15

Choose #15, Google Drive as Storage Option ** See help for drive backend at: https://rclone.org/drive/ **

Google Application Client Id

*Setting your own is recommended, see https://rclone.org/drive/#making-your-own-client-id for how to create your own.

If you leave this blank, it will use an internal key which is low performance.
I leave this blank in my tutorial

Client Secret

I also leave this blank in my tutorial

OAuth Client Secret

Leave blank normally.

Scope

1 Full access all files, excluding Application Data Folder. “drive”
2 Read-only access to file metadata and file contents. “drive.readonly” Access to files created by rclone only.
3 These are visible in the drive website. File authorization is revoked when the user deauthorizes the app. “drive.file” Allows read and write access to the Application Data folder.
4 This is not visible in the drive website. “drive.appfolder” Allows read-only access to file metadata but
5 does not allow any access to read or download file content. “drive.metadata.readonly”