Rclone - Rsync for cloud storage


Updated: July 28, 2017

Here's a bunch of interesting concepts. Rsync is the data moving workhorse of the Linux world, and I've blessed you with a detailed tutorial on this topic a while ago. Then, cloud is a new and brave concept where you can have your stuff saved to remote servers. You do this through a browser or a dedicated app.

But then, what happens when you have multiple cloud providers? The plot thickens and the complexity, uh well, biggens. Which is where programs like odrive come in play, except I never really got to master the finer side of things with this one. So, we need a browser, a dedicated app, and maybe command line? Aha. Enter Rclone.

Introduction & setup

Anyhow, Rclone is meant to provide a simple command-line interface for multi-cloud usage, with the added bonus of 'if it runs from the terminal, it can be scripted' kind of thing. Of course, this means you cannot just click your way through, so you do actually need to spend some time reading how it works and what it does. We'll get to that in more detail shortly.

Rclone can sync files and folders using Google Drive, Amazon S3, Dropbox, OneDrive, several other providers, and even your local filesystem, which makes it an effective drop-in for your generic rsync needs, too.

The setup is fairly simple. Download the archive and extract it somewhere, most likely your home directory, although this is not mandatory. Then, you will need to configure the tool.

rclone config

On first run, you will be presented with the following options:

./rclone config
2017/02/25 18:26:04 Config file "/home/roger/.rclone.conf" not found - using defaults
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q>

You can (or rather, must) configure new remotes - this actually means cloud services you wish to connect to and sync. The terminology is a rather ambiguous, but the official page does describe the usage model in more details, and fairly well. However, before you create new remotes, my suggestion is to set the configuration password.

This will encrypt the local database containing your cloud storage credentials, which can be quite important. After all, you do not keep your passwords in plain text on the disk in general, so there's no reason why Rclone should be an exception. However, do remember that if you forget the password, you will not be able to use Rclone, and you will need to start from scratch.

Your configuration is not encrypted.
If you add a password, you will protect your login information to cloud services.
a) Add Password
q) Quit to main menu
a/q> a
Enter NEW configuration password:
password:
Confirm NEW configuration password:
password:
Password set
Your configuration is encrypted.
c) Change Password
u) Unencrypt configuration
q) Quit to main menu
c/u/q>

Usage

I decided to test the functionality with OneDrive, especially since it's not a pure native Linux product per se, which makes it even more interesting. The official site has detailed instructions how to setup each service.

After I provided 9) Microsoft OneDrive as the storage type, Rclone launched the browser. At this point, I had to manually login into OneDrive and approve the utility as an authorized app, which let it download the necessary token for future remote logins.

OneDrive access, approve app

y/n> y
If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth
Log in and authorize rclone for access
Waiting for code...
Got code
2017/02/25 18:30:13 Failed to save new token in config file: section 'onedrive' not found
--------------------
[onedrive]
client_id =
client_secret =
token = {"access_token":"BzCCA3...

Current remotes:

Name                 Type
====                 ====
onedrive             onedrive

e) Edit existing remote
n) New remote
d) Delete remote
s) Set configuration password
q) Quit config
e/n/d/s/q>

I am not sure about the error 'Failed to save new token...', because it worked just fine, and I was able to use Rclone and communicate with OneDrive without any problems. This might be an innocent glitch, but it should be fixed, whichever way.

Using the program is quite simple. You specify commands you wish to use again remotes, where remotes are cloud storage accounts you've specified, e.g. onedrive, s3-new, etc. The syntax is relatively friendly, but it definitely can be made more obvious. I did use the simplistic placeholder value <remote> until I figured I had to replace it with whatever name of configured remotes. In our example, this is onedrive. AKA don't do this:

./rclone ls remote:
Enter configuration password:
password:
2017/02/25 18:33:03 Failed to create file system for "remote:": didn't find section in config file

File and folder listing, file copy, file sync, everything worked peachy. To be honest, I was having fun, there were no errors, and the software was behaving the way it should. Not bad for a tiny self-contained program that comes in a zip archive, without any extra bells or whistles.

./rclone lsd onedrive:
Enter configuration password:
password:
          -1 2017-02-16 00:50:30         3 Documents
          -1 2017-02-16 00:50:30         0 Pictures
          -1 2012-12-28 11:54:21         0 Public

./rclone ls onedrive:
Enter configuration password:
password:
   687149 smaker-first.docx
    58638 Documents/Document 1.docx
    10881 Documents/Document 2.docx
   753052 Documents/Document.docx

./rclone copy /home/roger/Pictures/Testing/ onedrive:testology
Enter configuration password:
password:
2017/02/25 18:35:33 One drive root 'testology': Waiting for checks to finish
2017/02/25 18:35:33 One drive root 'testology': Waiting for transfers to finish
2017/02/25 18:35:35
Transferred:   930.776 kBytes (186.655 kBytes/s)
Errors:                 0
Checks:                 0
Transferred:            4
Elapsed time:        4.9s

Successful copy

./rclone sync /home/roger/Pictures/Testing/ onedrive:testology
Enter configuration password:
password:
2017/02/25 18:38:37 One drive root 'testology': Waiting for checks to finish
2017/02/25 18:38:37 One drive root 'testology': Waiting for transfers to finish
2017/02/25 18:38:38 Waiting for deletions to finish
2017/02/25 18:38:38
Transferred:   865.962 kBytes (256.519 kBytes/s)
Errors:                 0
Checks:                 1
Transferred:            3
Elapsed time:        3.3s

./rclone check /home/roger/Pictures/Testing/ onedrive:testology
Enter configuration password:
password:
2017/02/25 18:39:16 One drive root 'testology': Building file list
2017/02/25 18:39:16 Local file system at /home/roger/Pictures/Testing: Building file list
2017/02/25 18:39:16 One drive root 'testology': 0 files not in Local file system at /home/roger/Pictures/Testing
2017/02/25 18:39:16 Local file system at /home/roger/Pictures/Testing: 0 files not in One drive root 'testology'
2017/02/25 18:39:16 One drive root 'testology': Waiting for checks to finish
2017/02/25 18:39:16 One drive root 'testology': 0 differences found

Security & problems

Overall, I was quite impressed how simple and intuitive the tool was, and I did not struggle much getting things done. Only one or two failed commands before it was all running smoothly. There were some niggles, and I do have a few recommendations on how everything can be made better.

Security wise, you need to provide your password, if you're using one, every time you launch the program, which means there's a level of interactiveness, and this breaks the idea of scripting and scheduled runs. You can work around this by using an environment variable RCLONE_CONFIG_PASS, but that effectively means storing your password for the tool somewhere on the disk.

In the interactive mode, as the official site suggests, you can create a file called set-rclone-password, which will prompt you for password from the command line and then set the correct environment variable. Namely, create a file with the following contents:

read -s RCLONE_CONFIG_PASS
export RCLONE_CONFIG_PASS

Then, source it:

source set-rclone-password

However, this is no different than manually doing:

export RCLONE_CONFIG_PASS=whatever

I think a much better way would be to encrypt the configuration anyway, but do it in a unique way that takes into account the computer signature, a combo of system info, disk identification, whatever, so that it cannot be executed on other machines just by copying the data. This way, the user would need not provide the credentials every single time, and the login info will still be securely stored.

The other problem is that Ctrl + C breaks/interrupts the execution. This may be cosmetic, but when copying & pasting text from a terminal window, you may accidentally disrupt the running process. Might be painful if you are copying big files.

Conclusion

I do have to say I'm pleased. I've always liked Rsync, I always found it very accurate and simple, and the Rclone implementation is no exception, creating for me, for the first-time, a real opportunity for a hassle-free management of multiple cloud platforms. This means it seems robust, it works well, it's intuitive, and I can create an automated backup mechanism, should I feel so inclined.

You may argue that all cloud provides do this seamlessly with their own utilities, but this is a centralized facility, very innocent, very simple and it works well in a world where Linux implementations are a token gesture to show that companies care, but then only come out with a half-beta product that is three generations behind the Windows counterpart. Choose any which software provider, not just cloud.

I believe the security mechanism can be streamlined, the built-in documentation can be clearer, but other than that, from my brief testing, Rclone looks like a nice little tool, and I will be exploring some more. The intuitive part makes it even more engaging. Well worth testing. P.S. Thanks to dogbite for this recommendation. Off you go, my little cloud dancers.

Cheers.

RSS Feed icon

del.icio.us del.icio.us stumbleupon stumble digg digg reddit reddit slashdot slashdot



Advertise!

Would you like to advertise your product/site on Dedoimedo?

Read more

Donate to Dedoimedo!

Do you want to
help me take early retirement? How about donating
some dinero to
Dedoimedo?

Read more

Donate