Skip to content

Rclone

Rclone (by Nick Craig-Wood) is "rsync for the cloud". Basically, it is used to transfer data to or from a variety of supported cloud storage providers (eg Google Drive).

Rclone is used by Cloudplow and Backup to upload media and backup Saltbox, respectively.

The guide below assumes you are using Google Drive.

Rclone supports many cloud provider backends, but the only one routinely used by the Saltbox team is Google Drive.

This process will use various scripts to do as much of this for you as possible, but there are some things that can't be scripted easily, like steps 1 and 2 below.

It also assumes you are using a Google Workspace account, since it assumes you can create shared drives. You can do some of this without a Workspace account, but the differences are not documented here. You won't be able to directly follow the steps below, and most of the scripts won't work for you.

Warning

IF YOU ARE HERE TO DO THIS A SECOND TIME, RETHINK THAT. IF YOU SUCCESSFULLY RAN THROUGH THIS PROCESS ONCE, YOU HAVE EVERYTHING YOU NEED TO SET SALTBOX UP AND SHOULD REUSE THOSE SHARED DRIVES, SERVICE ACCOUNTS, AND GROUP. THERE'S RARELY A REASON TO CREATE A SECOND SET.

If you already have Rclone configured, you can jump directly to the relevant section.

If you already have media on shared drives from your time with Cloudbox or PlexGuide or the like, you most likely DO NOT WANT TO DO THIS. This process is assuming you are starting from scratch without any of this already set up.

That said, let's proceed.

New Rclone Setup

Warning

YOU CANNOT SKIP STEPS HERE: EACH OF THESE STEPS IS ASSUMING YOU HAVE PERFORMED THE PREVIOUS ONE.

Info

IF YOU HAVE EXISTING GOOGLE DRIVES FROM ANOTHER CONTEXT [Cloudbox, PG, etc] USE THAT CONFIG [NOTABLY THE RCLONE CONFIG AND ANY SERVICE ACCOUNTS] IN A MIGRATION.

Warning

THIS PROCESS DOES NOT ACCOUNT FOR USING YOUR OWN TEAMDRIVES.

Step 1: Verify that the Shared Drive permissions are correct on your Google account

Detailed instructions here

Step 2: Create a new project and generate a credential file

Detailed instructions here

Save that credential file on your server at /opt/sa/project-creds.json

Step 3: Create a Google Group to hold service accounts

Detailed instructions here

Step 4: Set up the GCloud SDK

Detailed instructions here

Step 5: Generate a random prefix

Your randomly-generated prefix is:

10 RANDOM CHARACTERS SHOULD APPEAR HERE

That says '10 RANDOM CHARACTERS SHOULD APPEAR HERE'
Apparently the Javascript didn't work or you have Javascript disabled. Try reloading the page. If that doesn't work, generate it manually: [Type this at a command prompt on your server]
prefix=$(head /dev/urandom | tr -dc a-z | head -c10 ;) && echo $prefix

Make a note of that prefix; you will use it in the next two steps.

Info

There is nothing special about that prefix; it is ten random characters. It's not tied to you literally. When you reload this page, the prefix will change. That's fine. The specific prefix doesn't matter; just pick one and use it.

Why do I need this?
This prefix is used for two purposes:

1. Project names need to be unique across all of Google; a random prefix helps ensure this [the error that results in this case is non-obvious].

2. It helps these scripts unambiguously identify things that they have created, so they don't affect any projects, service accounts, or drives you may already have created.

Step 6: Generate 300 service accounts

Detailed instructions here

Detailed instructions here

Step 8: Verify that the union remote shows you the expected contents

Warning

IF YOU HAVE SKIPPED ANY OF THE PREVIOUS STEPS THIS VALIDATION WILL NOT WORK.

rclone tree google:/

This should display something like [the number and names of the files and folders may vary somewhat depending on your config]:

/
├── -- aZaSjsklaj-Movies Shared --
├── -- aZaSjsklaj-Music Shared --
├── -- aZaSjsklaj-TV Shared --
├── Media
│   ├── Movies
│   ├── Music
│   └── TV
├── azasjsklaj-movies_mounted.bin
├── azasjsklaj-music_mounted.bin
└── azasjsklaj-tv_mounted.bin

7 directories, 3 files
What if I don't see that?
If you see an error like this:
Failed to tree: 3 errors: aZaSjsklaj-Movies: failed to get Shared Drive info: googleapi: Error 404: Shared drive not found: BINGBANGBOING, notFound; aZaSjsklaj-Music: failed to get Shared Drive info: googleapi: Error 404: Shared drive not found: BANGBOINGBING, notFound; aZaSjsklaj-TV: failed to get Shared Drive info: googleapi: Error 404: Shared drive not found: BOINGBINGBANG, notFound
The most likely cause is that something went wrong in the group setup. Perhaps all the service accounts didn't get added to the group. Repeat the last part of [this step](google-group-setup.md) where you upload the members.csv and verify that the group shows at least 300 members after you're done.

You now have shared drives and union combining them; the saltbox install will merge this with your local drive and cloudplow will upload to the union mount, which will distribute media to the shared drives by path.

After the saltbox install

There is one thing you may wish to do after the saltbox install is complete.

You will still be limited to the 750GB/day Google upload limit until you configure cloudplow to upload directly to the individual shared drives. Eventually this will be automated, but for now there is this guide.

Go back to the install process.

Existing Rclone Setup

The default remote specified in settings.yml is google for Google Drive. If the Rclone remote in your config has the same name, then you are OK to skip this page and go on to the next.

If you are using Google Drive and the Rclone remote in your config has a different name, then you will need to either:

  • Rename your current Rclone remote to the default one (i.e. google). Instructions for this are below.

Or

  • Edit the Rclone remote entry in settings.yml to reflect yours.

If you prefer to use another cloud storage provider, you can add the name of the Rclone remote in to settings.yml.

Rename Existing Rclone Remote

To rename the Google Drive remote to google:

  1. Find and edit your Rclone configuration file.
nano $(rclone config file | tail -n 1)
  1. Rename the Google Drive drive remote (name between the brackets) to google.

  2. It will now look like this:

[google]
type = drive
client_id = JOHNNY.apps.googleusercontent.com
client_secret = JOEY
token = {"access_token":"ya30.DEEDEE-38ikRIxZvimyoxyKdse$
  1. Save the file and exit: Ctrl + X Y Enter.

  2. Copy the config file to ~/.config/rclone/rclone.conf (if it isn't there already):

cp -n $(rclone config file | tail -n 1) ~/.config/rclone/rclone.conf
  1. Give it the proper ownership and permissions. Replace user and group to match yours (see here):
sudo chown user:group ~/.config/rclone/rclone.conf
sudo chmod 755 ~/.config/rclone/rclone.conf