How to connect to external storage with rclone¶

Introduction¶

This guide describes how to access external storage using rclone. The emphasis is on the storage options provided by UZH. However, it should be easy to adapt the approach to other providers supported by rclone. UZH users have access to the following storage options.

SWITCHdrive¶

SWITCHdrive is based on an ownCloud instance and supports WebDAV protocol, which makes it accessible via rclone.

Swift Object Store¶

ScienceCloud offers Swift object store that can be accessed by rclone. See our how-to guide for more details

SMB file shares¶

An approach to mount an SMB share on a ScienceCloud instance is described in FAQs.

Additionally, for example from ScienceCluster, you can use smbclient.

For using rclone with smb, see here.

OneDrive (Office 365)¶

Application access to OneDrive is currently not enabled. It is not be possible to access it from a server even though rclone supports OneDrive as a backend.

SFTP (ssh)¶

With rclone you can list, copy, sync or mount a remote folder via SFTP, which uses your ssh authentication. This can be useful for sharing files between ScienceCloud and ScienceCluster. See rclone sftp for more details.

Installation¶

ScienceCloud¶

Most distributions have rclone in their official repositories. For example, on a ScienceCloud instance running Ubuntu you can install it with the following command.

sudo apt install rclone

ScienceCluster¶

To use rclone from the ScienceCluster, run the command module load rclone.

Configuration¶

See rclone config for full details.

SWITCHdrive¶

You can access SWITCHdrive using the WebDAV backend. Before you can configure an rclone remote, you would need to log in to your SWITCHdrive account to obtain the WebDAV URL and an application token.

To get the WebDAV URL, click on the cogwheel in the bottom right corner of the page and copy the URL from the text box that appears. Please make sure you copy the whole URL, which should look like https://drive.switch.ch/remote.php/dav/files/xxxx.xxxx%40uzh.ch/.

To generate an app token, click on your name in the upper right corner and select Settings. Click on Security in the left navigation bar and scroll to the very bottom of the page. In the App Name text field, enter the desired application name, for example remote. It is recommended to include the server information especially if you are using rclone on multiple machines. Finally click on the Create new app passcode button and copy the generated token. Please note that you will not be able to look up your token afterwards. However, you can always generate a replacement token if you forget this one.

To start the configuration of a remote, type rclone config on the server where you have previously installed rclone. This will launch the remote configuration wizard.

Enter n to create a new remote
Enter the desired remote name, e.g. switch
Enter webdav or the number corresponding to WebDAV from the list that appears
Enter the WebDAV URL you have retrieved earlier
Enter owncloud or the number corresponding to ownCloud from the list that appears
Enter your username that you use to log in to SWITCHdrive
Enter y to add the password to the configuration
Type or paste your app token
Type or paste your app token again for confirmation
Enter no to skip the advanced config
Review the remote config record and enter y to approve.
Enter q to exit the wizard

Important

While you can use your password instead of the app token, such usage is strongly discouraged.

Usage¶

See rclone subcommands for full details.

copy¶

You can copy files to and from the storage using the copy command, e.g.

rclone copy remote:Private/pictures/Paris.jpg data/mypictures
rclone copy /scratch/$USER/simulation/results.tar.gz remote:simulation

The first command copies Private/pictures/Paris.jpg from SWITCHdrive to your local directory data/mypictures. The second command copies /scratch/$USER/simulation/results.tar.gz to the simulation folder on SWITCHdrive.

You can also synchronise a whole directory. For instance, the following command copies all changes from your local data/results directory to archive folder on SWITCHdrive.

See rclone copy for more details.

sync¶

rclone sync data/results remote-switch:archive

Warning

rclone sync does not ask for a confirmation. It will silently delete from the target directory any files that are not present in the source directory. You need to be especially careful when using variables and expansions like . and ~ in your local paths.

You can find more information about available rclone commands online or by executing rclone command --help, e.g. rclone sync --help.

See rclone sync for more details.

mount¶

First, create a local folder to where rclone should mount.

mkdir -p mnt

Then, run the mount command

rclone mount remote: ~/mnt/

See rclone mount for more details.