The Best Tool You’re Not Using

10 min readFeb 21, 2019

Every day we jump from cloud to cloud and to me, it seems normal to have several cloud storage providers, an external drive, as well as some random files and folders on your desktop/laptop.

My work account has 120GB on GDrive, I have a 1TB Dropbox, a 4TB MEGA, and I own a domain email hosted on GSuite with over 800GB+ of files. That's a total of ~6TB across 4 platforms.

I need to come to term that I am a cloud storage addict.

One of the steps I have taken in getting my life online under control is to automate as much as I can. Moving files, syncing folders, updating versions, these are all tedious tasks much better suited to a computer program.

Say hello to Rclone. An offshoot of another open source program Rsync. Both extremely powerful tools that can be used to manage multiple directories, local files and remote, to make your life a little bit easier. As a command line tool, there is a learning curve and understanding what you are typing is essential to make sure files are not duplicated and overwritten. Below is a starter’s guide to Rclone based on my own learning. I am not the expert, and Reddit has introduced me to many talented and smart(er) people who have done amazing things with these tools.

Below are the steps you need to take to use this amazing tool, broken down so that you understand exactly what you are doing and how to do it. Below I highlight some of the key features of Rclone that make it an amazing tool.

Step 1

Download it. The Rclone site breaks this down very well and if you are on MAC/Linux you don’t even need to visit the site. Simply open up a Terminal window (on a mac) and paste in:

curl https://rclone.org/install.sh | sudo bash

For a windows machine, check out the downloads page. If you have a semi-recent computer running Windows 10, you will download and launch the AMD64 Windows edition (As of writing Rclone is on v.1.46).

Step 2

Once Rclone is installed, we can begin to check out the features. I am using a MacBook so all screenshots are from Terminal. We start Rclone by running:

rclone config

From here we are greeted with a simple menu. Initially, I would always suggest creating a configuration password. If someone has access to your computer, and they are aware of this program they could sync files across remotes to any location their evil mind chooses. To do this type “s” and follow the on-screen instructions.

Step 3

It’s time to create your remotes. Rclone has integrations with many of the major and minor cloud storage providers and the interface guides you through the steps. There are traditional integrations and some other useful encrypted and mounted options also. For my demo, i’ll be syncing files from my Dropbox to my GDrive.

1Fichier
Alibaba Cloud (Aliyun) Object Storage System (OSS)
Amazon Drive (See note)
Amazon S3
Backblaze B2
Box
Ceph
Citrix ShareFile
C14
DigitalOcean Spaces
Dreamhost
Dropbox (Referal link)
FTP
Google Cloud Storage
Google Drive (Referral Link for 20% off)
Google Photos
HTTP
Hubic
Jottacloud
IBM COS S3
Koofr
Mail.ru Cloud
Memset Memstore
Mega
Microsoft Azure Blob Storage
Microsoft OneDrive
Minio
Nextcloud
OVH
OpenDrive
Openstack Swift
Oracle Cloud Storage
ownCloud
pCloud
premiumize.me
put.io
QingStor
Rackspace Cloud Files
rsync.net
Scaleway
SFTP
Wasabi
WebDAV
Yandex Disk
The local filesystem

Pick your remote by typing either the name of the service “dropbox” in my case. From here we are guided through the setup of each remote. Rclone have fantastic docs on the setup of each remote and they do example walkthroughs on each.

Dropbox Walkthrough

GDrive Walkthrough

For most remotes, you can limit Rclone’s access to Read/Write in several ways. Option 1 and 2 are the most used options and will liekly be the only options required. Ideally to limit files being lost, the source remote should be set up with Read-Only (2) and the Destination with Read-Write (1).

Additionally, the advanced config is rarely required and auto config will work is 99% of cases. If you are setting up a migration server or have specific requirements, Rclone docs cover the details better than I could.

If your machine is local and not on a server, select “Y” for auto-config and a new browser window will open allowing Rclone to access your GDrive. Accept the permissions and copy and paste the token into your terminal. Unlikely unless you’re setting up a mass migration with multiple accounts that you need a TeamDrive, and there is rarely any need to edit the config file.

Once you have set up 2 (or more) remotes, we can begin to test out some file transfers.

Step 4

Below I have set up 2 remotes, their names can be generic or unique. From here we quit (q) and we can begin to utilize the exploratory features of Rclone. Exit the config (q) and test your remote setup.

Note 1: Likely if you are unable to list files or have ERROR messages, you may have set up your remote drives incorrectly in Rclone. If you have done this, go to “rclone config”, select the “Delete remote — d” option and remove the remote to start again.

Note 2: When accessing a drive, some remotes take a few moments to establish a connection. Don’t worry if you are waiting 20–30 seconds for the remote to respond.

Note 3: If you need to see what is happening for any command in Rclone. Add “-vv” (Very detailed) or “-v” (Normal detail) to the end of the command, this will generate messages and display any errors. Errors are normal, for example when accessing GDrive, you will see many errors like “error googleapi: Error 403: Rate Limit Exceeded”. This is normal and Rclone handles these errors for you.

Some example commands used to explore folder and files in Rclone:

rclone lsd [remote name]:/[Optional - Specify Remote Folder](List the directories of the top level folder or specific folder)rclone ls [remote name]:/[Optional - Specify Remote Folder] 
(List all the files in the remote or specific folder)rclone lsl [remote name]:/[Optional - Specify Remote Folder] 
(List location / size / of all files in remote or specific folder)

All the commands are listed in the Docs here.

Note above 2 things:

Running the ls command on the whole remote will produce a long list of files, it’s better to specify a specific folder and run the command there.
If your folder has spaces in its name, use quotation markets to bind it together. Example: Temp folder → ‘Temp Folder’

Step 5

Moving files and folders is easy once you understand the structure. I have set up servers on providers like Hetzner where once SSH’d in, I setup Rclone and move 800GB at a time using their Gigabit speeds to transfer files between services. In another post, I’ll show you how to set this up and I’ll link it here.

Below are the steps required to transfer files between 2 providers, GDrive and Dropbox. Remember to be careful when using some of these commands as they can be destructive (Files get deleted or removed). I can’t help if you move files and they get lost or deleted.

The most basic command is:

rclone copy [Source]:/[Folder] [Destination)]:/[Folder]

The above command is like a multitool. It does a lot of stuff but it doesn’t do anything as good as a dedicated tool would. We will break it down:

rclone copy <-- This is the copy command that Copy files from source to dest, skipping already copied. Use this one for first sync as files won’t be deleted when used.[Source]:/[Folder] <-- Source is the remote containing the files. And if you are only syncing a specific folder, then ensure you specify this folder or you will sync the whole remote.[Destination)]:/[Folder] <-- Destination is the remote recieving the files. And if you are putting the files in specific folder, then ensure you specify this folder or you end up with a messy remote.

What's happening above:

Checking top-level folders in dropbox:
Checking contents of dropbox:/‘Test Folder’
Checking top-level folders in gdrive:
Checking contents of ‘Test’
Copying Files from dropbox:/’Test Folder’ to gdrive:/Test
Checking contents of dropbox:/‘Test Folder’
Checking contents of gdrive:/‘Test’

You can see that the files are mirrored from my Dropbox to my GDrive. Without Rclone I’d download the files and upload them, creating 3 copies of the files (1xDropbox, 1xLocal, 1xGDrive). This way there is 1 copy mirrored across 2 Remotes. Normally I don't check every folder but to show the transfer I am breaking it down.

Note: A bonus to using Rclone is that file metadata is preserved. If a file is dated in the past on Remote1, the same date is seen on Remote2. Very handy when transferring large sets of files.

The most basic command is:

rclone sync [Source]:/[Folder] [Destination)]:/[Folder]

The above command is using: sync. A similar command to copy but this command can be destructive. We will break it down with an example:

Now let’s try the same command using the “sync” option.

What’s happening:

List all the files in dropbox:/’Test Folder’
Delete the file: 123.png
List all the files in dropbox:/’Test Folder’ (Showing 123.png is gone)
List all the files in gdrive:/’Test’ (Showing 123.png is present)
Sync the 2 folders across the two remotes.
List all the files in dropbox:/’Test Folder’ (123.png still deleted)
List all the files in gdrive:/’Test’ (Showing 123.png is now gone)

Above, we can see the destructive use of SYNC. If we want to exactly mirror 2 remotes, this can be very useful. If we want to backup 2 remotes this is not useful as all changes are copied from the source remote to the destination remote.

Bonus Step — Flags

Getting fine control of Rclone can unlock many benifits such as increased speed, updating, and more. Below are some of my favourites broken down:

rclone copy [Source]: [Dest]: -v -c --bwlimit 8M --transfers=3 --stats=30s --dry-run-v <-- Using the -v flag, rclone will produce Error, Notice and Info messages.-c <-- Rclone will look at modification time and size of files to see if they are equal. If you set this flag then rclone will check the file hash and size to determine if files are equal.--bwlimit 8M <-- Taking up all the bandwidth is rude. I fif you need to limit the speed of the transfers use this flag. Example: GDrive limits daily transfers to 750GB, or ~9MB/s.--transfers=3 <-- Specify the number of files being transferred in parallal. The default is 4, but I have gone up to 20 without any issues.--stats=30s <-- Rclone will produce an update stat every X seconds or minutes to show you the rate or transfer or other key details.--dry-run <-- Before commiting to a transfer and potentially regretting it, run the command with this flag. It will show you what will happen when you run a copy/sync/etc without performing it. One you are confident, remove this flag and repeat the command.

Some more useful flags:

Update Remotes:

--update <-- Rclone will be forced to skip any files which exist on the destination and have a modified time that is newer than the source file.

Progress Block:

--progress <-- Rclone update the stats in a static block in the terminal providing a realtime overview of the transfer.

Immutable

--immutable <-- Rclone treat the source and destination files as immutable and won't allow any modification. This does NOT stop files being deleted, existing files will only never be updated.

Conclusion

Rclone is an amazing tool. Once you understand the structure of the command and know how to navigate the Docs, you can automate, clean, encrypt, delete, and more. It has saved me a lot of time and headache.

I enjoyed writing this, as my first article it was something that I spent a lot of time on. I want to cover Rclone and Rsync in detail as there are so many useful features that can be used to save time (and money). For context, my current setup looks something like this:

My next topic I will be covering Rclone crypt and also showing you how to mount the remote locally to use your computer’s GUI to navigate files.

Eventually, I will cover automating scripts on a daily/weekly basis from a VPS. Essentially automating the whole process end-2-end.

Thanks @all,

David.