Skip to content

How to connect to external storage with rclone

Introduction

This guide describes how to access external storage using rclone. The emphasis is on the storage options provided by UZH. However, it should be easy to adapt the approach to other providers supported by rclone. UZH users have access to the following storage options.

SWITCHdrive

SWITCHdrive is based on an ownCloud instance and supports WebDAV protocol, which makes it accessible via rclone.

Swift Object Store

ScienceCloud offers Swift object store that can be accessed by rclone. See our how-to guide for more details

SMB file shares

An approach to mount an SMB share on a ScienceCloud instance is described in FAQs.

Additionally, for example from ScienceCluster, you can use smbclient.

For using rclone with smb, see here.

OneDrive (Office 365)

Application access to OneDrive is currently not enabled. It is not be possible to access it from a server even though rclone supports OneDrive as a backend.

SFTP (ssh)

With rclone you can list, copy, sync or mount a remote folder via SFTP, which uses your ssh authentication. This can be useful for sharing files between ScienceCloud and ScienceCluster. See rclone sftp for more details.

Installation

ScienceCloud

Most distributions have rclone in their official repositories. For example, on a ScienceCloud instance running Ubuntu you can install it with the following command.

sudo apt install rclone

ScienceCluster

To use rclone from the ScienceCluster, run the command module load rclone.

Configuration

See rclone config for full details.

SWITCHdrive

You can access SWITCHdrive using the WebDAV backend. Before you can configure an rclone remote, you would need to log in to your SWITCHdrive account to obtain the WebDAV URL and an application token.

To get the WebDAV URL, click on the cogwheel in the bottom right corner of the page and copy the URL from the text box that appears. Please make sure you copy the whole URL, which should look like https://drive.switch.ch/remote.php/dav/files/xxxx.xxxx%40uzh.ch/.

To generate an app token, click on your name in the upper right corner and select Settings. Click on Security in the left navigation bar and scroll to the very bottom of the page. In the App Name text field, enter the desired application name, for example remote. It is recommended to include the server information especially if you are using rclone on multiple machines. Finally click on the Create new app passcode button and copy the generated token. Please note that you will not be able to look up your token afterwards. However, you can always generate a replacement token if you forget this one.

To start the configuration of a remote, type rclone config on the server where you have previously installed rclone. This will launch the remote configuration wizard.

  • Enter n to create a new remote
  • Enter the desired remote name, e.g. switch
  • Enter webdav or the number corresponding to WebDAV from the list that appears
  • Enter the WebDAV URL you have retrieved earlier
  • Enter owncloud or the number corresponding to ownCloud from the list that appears
  • Enter your username that you use to log in to SWITCHdrive
  • Enter y to add the password to the configuration
  • Type or paste your app token
  • Type or paste your app token again for confirmation
  • Enter no to skip the advanced config
  • Review the remote config record and enter y to approve.
  • Enter q to exit the wizard

Important

While you can use your password instead of the app token, such usage is strongly discouraged.

Usage

See rclone subcommands for full details.

copy

You can copy files to and from the storage using the copy command, e.g.

rclone copy remote:Private/pictures/Paris.jpg data/mypictures
rclone copy /scratch/$USER/simulation/results.tar.gz remote:simulation

The first command copies Private/pictures/Paris.jpg from SWITCHdrive to your local directory data/mypictures. The second command copies /scratch/$USER/simulation/results.tar.gz to the simulation folder on SWITCHdrive.

You can also synchronise a whole directory. For instance, the following command copies all changes from your local data/results directory to archive folder on SWITCHdrive.

See rclone copy for more details.

sync

rclone sync data/results remote-switch:archive

Warning

rclone sync does not ask for a confirmation. It will silently delete from the target directory any files that are not present in the source directory. You need to be especially careful when using variables and expansions like . and ~ in your local paths.

You can find more information about available rclone commands online or by executing rclone command --help, e.g. rclone sync --help.

See rclone sync for more details.

mount

First, create a local folder to where rclone should mount.

mkdir -p mnt

Then, run the mount command

rclone mount remote: ~/mnt/

See rclone mount for more details.