How to use SWIFT object store¶

SWIFT 3 layers architecture¶

Account layer¶

Each project in ScienceCloud owns an individual SWIFT account and quota. So at the very beginning, you need to ask for the SWIFT quota by contacting Science IT. Once your SWIFT quota is raised you will be able to create a container under Object Store panel of ScienceCloud dashboard.

Container layer¶

Is another layer of abstraction introduced in SWIFT object store architecture which allows you the possibility to assign different storage policy on a per-container basis. Containers are composed of the list of your objects and can be considered as a structural layer.

Objects¶

Objects are your files that are stored as binary files along with their metadata as a single unit on the filesystem.

Swift has a 5 GB limit on the size of a single uploaded object. However files larger than 5 GB can be dealt with segmenting them and uploading the resulting segments with a special manifest file that, when downloaded, sends all the segments concatenated as a single object. Read more here.

Storage policies¶

ScienceCloud exposes  two storage policy in order to retain your data in case of failures. A storage policy can only be assigned when a container is created and once it is defined all objects will be stored in accordance to that policy.

replica-2¶

It is the default policy, objects are stored on two separate disks each container in a distinct server. This policy guarantees read/write access to the data in the event of a node failure or during maintenance periods.

ec104 (Erasure coding)¶

This policy reduces the amount of space used on the system splitting each object in 10 shards and computing 4 additional shards to allow for recovery. Each shards is then stored on a different disk and spread among the two SWIFT servers. This policy protects your data in case of disk failures (up to 3 disks hosting one of a shards of your object can fail at any given time without incurring in data loss), however your data will not be available until the system has been restored following a hardware failure or a scheduled maintenance.

Swift Roles¶

Swift Operator¶

Swift operator is a privileged role, introduced in 2016, and usually granted a subset of users in each project. The SwiftOperator user has the following access and rights within a project:

1. List the containers
2. Create/Delete a container
3. Update the meta information on containers
4. Set the read/write Access-Control List (ACL) on each container

Regular user¶

The non privileged users are able to:

1. Download objects from container whose the read access is granted
2. upload files or directories to the container with write access
3. Issue the temporary URL for the container
4. Obtain the statistics information for the container

But UNABLE to:

1. List the containers in the project
2. Set/remove ACL on containers

Available tools¶

There are several choices for managing your files within the SWIFT object storage which you can pick according to your liking and requisites:

OpenStack Swift tool¶

Swift client is the native command line utility for interacting with an OpenStack object storage environment. Although it lacks a graphical interface, it is the more popular and powerful tool for managing the data in object store. 

Setup Environment¶

Install the Swift python CLI and the keystone client (needed to authenticate on ScienceCloud) by issuing below cmds on your terminal:

pip install python-swiftclient
pip install python-keystoneclient

In order to setup your environment, please download Openstack RC file from ScienceCloud GUI by:  Access & Security > API Access  > Download OpenStack RC file.

Then import this variables into your shell environment by:

source /path/to/downloaded/rc_file.sh

Manage containers¶

To manage the containers some operations needs SwiftOperator while others have no need for the operator role enough to be a member of the project.

SwiftOperator role operations¶

List containers/objects¶

Basically gives you the containers list and, if called against a container, the list of objects within:

swift list
swift list container_name

Create a container with the default storage policy¶

To create a new container with the default storage policy (i.e. replica-2) you can simply do:

swift post <container_name>
swift post test_container

Create a container with the ec104 (Erasure Coding) storage policy¶

If you are aware of the downsides (see above) but would like to leverage the smaller storage overhead the ec104 storage policy has, you can create a container by:

swift post <container_name> -H 'X-Storage-Policy: <policy-name>'
swift post test_container -H 'X-Storage-Policy: ec104'

It is also advisable to allocate the _segments container that will be used by Swift when object larger than 5 GB are uploaded (failing to do so will result in a container to be automatically created with the default replica-2 policy)

swift post <container_name>_segments -H 'X-Storage-Policy: <policy-name>
swift post test_container_segments -H 'X-Storage-Policy: ec104'

Set access list¶

Swift post is the right command for setting up the container ACL. '-r'/'-w' option is used for read/write ACL respectively. Then we need to specify the project name and  the target user to be granted  access. Otherwise using '*' in any place is mend for 'any' user or project.

swift post -r 'project_name:shortname' <container_name>

Give the write access only to hrajab, and read right to everyone in project s3it.playground to container test.¶

swift post -r 's3it.playground:*' test
swift post -w 's3it.playground:hrajab' test

To delete the ACL just use the empty ''¶

swift post -r '' <container_name>
swift post -w '' <container_name>
swift stat test

Show the metadata for the container test¶

Create a public container¶

Public container is a public repository available to be read by everyone. The '.r,.rlistings'  are the right option for this purpose.

swift post -r '.r:*,.rlistings' <container_name>

The ".r:*" will remove all restrictions on reading. The ".rlistings" will allow also listing.

If you want to allow web listing of a container (so that you can easily access it with a browser) set the 'web-listings' meta key:

swift post -m "web-listings: true" <container_name>

Member role operations¶

Swift copy/download/upload¶

To download

# Example download
swift download <container_name> <object_name>
swift download test_container myobject.txt

# Example upload
swift upload <container_name> <path/to/object> --object-name <desired_object_name>
swift upload test_container ~/myobject.txt --object-name myobject.txt

# Example copy
swift copy <container_name> <object_name> -d </path/to/target_container>
swift copy test_container myobject.txt -d /target_container

Swift delete¶

Delete a container or objects within a container.

swift delete <container_name> <object_name>
swift delete test text.txt

Swift stat¶

It gives you more info such as number of objects, the size, Read/Write ACL and, policy storage for given container.

swift stat <container>
swift stat test_container
         Account: AUTH_6169dd64a7ed498581cee5abff0c6cd7
       Container: test
         Objects: 0
           Bytes: 0
        Read ACL:
       Write ACL:
         Sync To:
        Sync Key:
   Accept-Ranges: bytes
X-Storage-Policy: replica-2
     X-Timestamp: 1479479622.50440
      X-Trans-Id: txb88a0f4c9128426e9a11f-00584fe7b7
    Content-Type: text/plain; charset=utf-8

Where Account point your project path.

Swift tempurl¶

If you need to give a temporary access to your Swift object to whom is not part of your project, generating a temporary URL would be the right solution.

swift tempurl <method> <seconds> <path> <key>
swift tempurl GET 86400 /v1/AUTH_6169dd64a7ed498581cee5abff0c6cd7/test mykey
/v1/AUTH_6169dd64a7ed498581cee5abff0c6cd7/test?temp_url_sig=4a741a28ec93ee75d86d233a9e50b87282a9e15c&temp_url_expires=1481717953

This link is valid for 24 hours download.

rclone¶

A powerful command line program to copy and sync files and directories to and from SWIFT object store. This tool has some outstanding features such partial syncs, copy only changed file, check the files integrity and retain timestamps of each file that make it the preferred tool for managing large amount of data.

Rclone is a Go program that comes as a single binary file. Please see rclone downloads, download and run it by following the instructions.

Setup Environment¶

To setup a new remote SWIFT object store in rclone, there are two options:

An interactive configuration session which prompt you for the initial authentication params:

Open your terminal and enter: rclone config and enter 'n' for new remote

1. name> choose a name for this remote
2. storage> choose "Openstack Swift".
3. user> ScienceCloud username
4. key> ScienceCloud password
5. auth> https://cloud.s3it.uzh.ch:5000/v2.0
6. domain> default
7. tenant> your ScienceCloud project name
8. tenant_domain> default
9. region> (leave empty)

10. storage_url> (leave empty)

11. auth_version> (leave empty)

Confirm the configuration and Quit

As an alternative, if you already downloaded the Openstack RC file, you can use rclone by simply appending to it:

export RCLONE_CONFIG_MYREMOTE_TYPE=swift

export RCLONE_CONFIG_MYREMOTE_ENV_AUTH=true

and source it as usual: in this case the name of the rclone remote will default to "myremote:"

Using this approach you will always need to source your RC file before using rclone but your Webpass will not be saved to disk.

Manage containers¶

SwiftOperator role operations¶

Here we will walk you through a very brief but must know example of this tool, however, for the further and complete guide please refer to the rclone documentation.

Create a container¶

To create container e.g input_files in ScienceCloud:

rclone mkdir remote_name:<container_name>
rclone mkdir myremote:test_container

To remove a container "rclone rmdir" will do the job.

List containers¶

To list all the containers created on your Object store:

rclone lsd remote:path
rclone lsd sciencecloud:/

Member role operations¶

rclone copy/sync¶

If a user has been given read/write access to a specific container, the user will be able to copy and sync from/to the container.

rclone copy source:path dest:path
rclone copy test.txt myremote:/test_container

Copy test.txt from your current working directory to ScienceCloud into test_container .

rclone sync source:path dest:path
rclone sync /path/to/folder myremote:/test_container

Will sync your local folder with the destination folder modifying only the destination.

rclone mount¶

rclone includes the ability to mount a Swift container as a local filesystem.

What follows in an example of how to mount it for simple download/upload of files (editing of files currently on swift will not work)

rclone mount --vfs-cache-mode off my_remote:/ /mnt/

There are several caching strategies that can be selected to make it behave more similarly to a regular filesystem:

we suggest you to read https://rclone.org/commands/rclone_mount/#file-caching for more information.

unmount¶

The Swift container should be automatically unmounted with you type Ctrl+C. If that doesn't work, you may need to manually unmount the drive:

Linux¶

fusermount -u /path/to/local/mount

Mac¶

umount /path/to/local/mount

More details are here.

Special instructions for macOS¶

Homebrew can be used to install rclone.

brew install rclone

To use rclone mount on a Mac, you'll need to additionally install macFUSE. A reboot is necessary after installing macFUSE.

brew install --cask macfuse

If the mount fails, you may need to manually allow macfuse by going to  "System Preferences → Security & Privacy → General", authenticate as an admin by clicking the 🔒, and then click "Allow" to enable kernel extensions. More details are here and here.

Additional commands¶

There are also some other useful commands such as:

• rclone delete - To remove the contents of path (selective deleting).

• rclone purge - Remove a container and all of its contents.

• rclone check - Checks the files in the source and destination match.

• rclone ls - List all the objects in the the path with size and path.

• rclone lsl - List all the objects path with modification time, size and path

Cyberduck¶

Offers a graphical user interface and it's only supported by Windows and MacOS.

Setup connection¶

Check the following page: How to use CyberDuck to access SWIFT

Manage containers¶

SwiftOperator role operations¶

Create a container¶

On your first login, you will need to create at least one container (folder) to put your content in.

Choose File > New Folder and specify a name. There is only "RegionOne" available on ScienceCloud.

After the container has been created, you may start adding your content to the storage platform.

Member role operations¶

Upload/Download/Syncing¶

of the content is indeed straight forward by right clicking on the container and choosing:

1. Upload: will prompt you to pick the file for upload
2. Download: start downloading right away. To change the landing folder or file name, choose Download To or Download As.
3. Synchronise: To keep a directory tree sync from SWIFT storage to your local host, choose Synchronize and specify the directory when you prompted for.
4. Delete: you are able to delete an object by right-click on the object and Delete.

In addition,  bandwidth and number of concurrent connections can be limited using the toggle in the lower right of the transfer window.

Temporary URL¶

A private object stored in your container can be made publicly available for a limited time using a signed URL. This URL grant access to anyone to download the object, however, it also includes a date and time after which the URL will no longer work. Copy the signed URL by right-click on object and Copy URL.