Sync CLIent ‘Magic’

Pulling back the curtain and getting our hands dirty with the odrive CLI

Crack open that OS shell and take a big whiff — Ahhh. Smells like power and possibility, doesn’t it? Okay … maybe a little bit like Grandma’s house, too. The CLI is one of the oldest computer user interfaces, after all. Don’t let that crusty, blinking cursor chase you away, though. This thing is an oldie, but a goodie.

A CLI (or Command Line Interface) can initially appear cryptic, obtuse, and even a bit scary, but once you get through those hurdles you will be rewarded with an interface that is highly expressive and very flexible. Drop odrive’s Progressive Sync engine into the mix and things get really interesting.

The first incarnation of an odrive CLI was actually exclusively for the odrive Agent (Linux users: this is what you want to check out). Agent is a stand-alone, headless, scriptable version of the odrive Progressive Sync Engine. When we saw the capabilities that it afforded odrivers, we naturally wanted to extend those to our desktop sync client — and so we have!

Why use a CLI?

Have you ever wanted to tweak the behavior of an application to suit your unique needs? How about automating tasks to streamline your data management and workflow?

As convenient as GUIs (Graphical User Interfaces) are, they tend to cater to the most general use cases. Trying to implement every possible permutation, every conceivable user action, would take an enormous amount of time and you would probably end up with an absolute disaster of an interface.

With a CLI and a little creativity, you can expand and extend the capabilities of an application drastically. Work on things at a more granular level, behind the scenes. In the case of odrive’s sync, you can start pulling off some complex maneuvers with just a few well-placed commands.

Here are a few examples:

• Sync a file, modify it, and then unsync it
• Unsync files by size, date, or extension
• Run sync only at certain times to manage bandwidth use.
• Limit your local cached files to specific file types
• Search for all files of a certain type residing on your computer or linked clouds and move/copy them into other storage services for syncing.
• Display a notification when a new file is added to an odrive folder
• Test performance of your linked storage

Now, toss that mouse in the trash, let your keyboard take point, and lets dig in!

*Note: If you aren’t interested in scripting right now and just want to quickly get the Agent/CLI installed for using basic sync functionality, our documentation can walk you through that.*

*WARNING: TECHNICAL STUFF AHEAD*

The Basics

The odrive CLI is driven by a set of basic commands. These can be listed by issuing a -h or -help to the odrive CLI client.

authenticate        authenticate odrive with an auth key
mount               mount remote odrive path to a local folder
unmount             remove a mount
sync                sync a placeholder [--recursive] [--nodownload]
stream              stream placholder/remote file
refresh             refresh a folder
unsync              unsync a file or a folder
xlthreshold         split files larger than this threshold
encpassphrase       specify a passphrase for Encryptor folders
syncstate           get sync status info
status              get status info
deauthorize         deauthorize to unlink the current user and exit
diagnostics         generate diagnostics
emptytrash          empty odrive trash
restoretrash        restore odrive trash
shutdown            shutdown odrive

*Since we are constantly updating, the above commands are subject to change. Always issue the ‘-help’ command to see the latest and greatest. You can also check them out in our documentation*

As you can see, there are a variety of commands at your disposal. Primary among them are sync, refresh, unsync, and syncstate. With these four commands you can create a solution for almost any sync use case.

We’ll take the first of our examples above: Sync a file, modify it, and then unsync it.

Let’s say we want to add an entry to a simple text-based to-do list we store in the cloud. Now, I realize this may be a little bit of a silly use case since we can easily do this through the standard desktop interface, but this is just for demonstration. This example will show how we can use our core CLI commands to easily push a file through several of our primary sync states.

Below I have added some very basic scripts in both bash (OS X) and powershell (Windows).

**A note about these scripts: odrive.py is included with the odrive desktop client install. You will find it located in these paths, but you can move it to wherever you like:

Windows:
$HOME\.odrive\bin\[version#]\cli\odrive.py
Mac:
$HOME/.odrive/bin/[version#]/odrive.py

I know there are many folks who may not have Python installed or prefer a binary client. We have binary CLI clients available for download here. Use them in place of “python odrive.py” in the example scripts.**

Bash

F="$HOME\odrive\ACD\to-do.txt"
odrive_py="$HOME/.odrive/bin/5491/odrive.py"
python "$odrive_py" sync "$F.cloud"
while [[ $(python "$odrive_py" syncstate "$F") != 'Synced' ]]; do
   sleep 2
done
NEW_ENTRY="Write killer blog post for new odrive CLI"
echo $NEW_ENTRY >> "$F"
python "$odrive_py" refresh $(dirname "$F") >/dev/null
while [[ $(python "$odrive_py" syncstate "$F") != 'Synced' ]]; do
   sleep 2
done
python "$odrive_py" unsync "$F"

Powershell

$F = "$HOME/odrive/ACD/to-do.txt"
$odrive_py = "$HOME\.odrive\bin\5453\cli\odrive.py"
python "$odrive_py" sync "$($F).cloud"
while ($(python "$odrive_py" syncstate "$F") -ne 'Synced') {
   sleep 2
}
$NEW_ENTRY = "Write killer blog post for odrive CLI - Windows"
echo $NEW_ENTRY >> "$F"
python "$odrive_py" refresh $((get-item "$F").Directory.FullName) > $NULL
while ($(python "$odrive_py" syncstate "$F") -ne 'Synced') {
   sleep 2
}
python "$odrive_py" unsync "$F"

The logic for both scripts is exactly the same, with only minor syntax differences for the two shells. Here is an explanation, line-by-line:

• Set the file name that we wish to edit ($F).
• Sync the file using the CLI command “sync”. We append .cloud to the file name because the file is starting in an unsynced (placeholder) state.
• Loop until the file has been synced down locally, sleeping for 2 seconds on each pass. For this check we use the CLI command “syncstate”, looking for the string ‘Synced’ as the return.
• Once the file is synced, append our new to-do entry to the end of it
• Perform a sync refresh on the parent folder of our file, using the CLI command “refresh”. This will ensure that odrive picks up the change immediately and starts uploading it to the remote storage.
• Loop until the file has finished syncing, sleeping for 2 seconds on each pass. We, again, use the CLI command “syncstate”, looking for the string ‘Synced’ as the return.
• Once the file has been successfully synced, we unsync using the CLI command “unsync”.

**For the sake of a clearer example, I statically defined the file name ($F) and the new to-do list entry ($NEW_ENTRY), but it would be more practical to pass these values into the script for “real-world” use.**

Advanced Scripting

Still with me?

Let’s go through one more example: Unsync files by size, date, or extension

This is a more advanced script and accepts inputs from the user. We’ll be using the CLI command “unsync” here.

Bash

Usage: unsync_by [-e <extension>] [-s <size in kilobytes>] [-d <days>] [directory path] [-r]
Help: Unsync files by extension, size exceeded, or days old for a given directory
Options:
-e, --extension Unsyncs files with the specified extension
-s, --size Unsyncs files larger than the specified size in kilobytes
-d, --days Unsyncs files older than the specified day
-r, --recursive Unsyncs files recursively through the specified path
-h, --help Help

#!/bin/bash
set -e
ODRIVE_PY="$HOME/temp/odrive python/odrive.py"
RECURSIVE="-maxdepth 1"
if [[ $1 == "-h" ]] || [[ $1 == "--help" ]] ; then
        echo "Usage: unsync_by [-e <extension>] [-s <size in kilobytes>] [-d <days>] [directory path] [-r]"
        echo "Help: Unsync files by extension, size, or days old for a given directory"
        echo "Options:"
        echo "-e --extension  Unsyncs files with the specified extension"
        echo "-s --size       Unsyncs files larger than the specified size in kilobytes"
        echo "-d --days       Unsyncs files older than the specified day"
 echo "-r --recursive  Unsyncs files recursively through the specified path"
        echo "-h --help       Help"
elif [[ $# -gt 2 ]] ; then
    key="$1"
    case $key in
        -e|--extension)
        EXTENSION="$2"
        ;;
        -s|--size)
        SIZE="$2"
        ;;
        -d|--days)
        DAYS="$2"
        ;;
 *)
        echo "Invalid arguments. Please consult help for usage details (-h, --help)."
     exit 1
 ;;
     esac
else
     echo "Invalid arguments. Please consult help for usage details (-h, --help)."
     exit 1
fi
DIRPATH="$3"
if [[ $4 == "-r" ]] || [[ $4 == "--recursive" ]] ; then
     RECURSIVE=""
fi
if [[ $EXTENSION ]] ; then
     echo unsyncing all files of "$EXTENSION" type in "$DIRPATH"
     find "$DIRPATH" $RECURSIVE -name "*$EXTENSION" -type f -exec python "$ODRIVE_PY" unsync {} \;
elif [[ $SIZE ]] ; then
     echo unsyncing all files larger than "$SIZE" kilobytes in "$DIRPATH"
     find "$DIRPATH" $RECURSIVE -size "+${SIZE}k" -type f -exec python "$ODRIVE_PY" unsync {} \;
elif [[ $DAYS ]] ; then
     echo unsyncing all files older than "$DAYS" days in "$DIRPATH"
     find "$DIRPATH" $RECURSIVE -mtime "+${DAYS}" -type f -exec python "$ODRIVE_PY" unsync {} \;
fi

Powershell

Usage: unsync_by [-e <extension>] [-s <size in kilobytes>] [-d <days>] [-p <directory path>] [-r]
Help: Unsync files by extension, size exceeded, or days old for a given directory
Options:
-e, -extension Unsyncs files with the specified extension
-s, -size Unsyncs files larger than the specified size in kilobytes
-d, -days Unsyncs files older than the specified day
-p, -dirpath The specified path
-r, -recursive Unsyncs files recursively through the specified path
-h, -help Help

Param(
   [parameter()][alias("e")][string]$EXTENSION,
   [parameter()][alias("s")][string]$SIZE,
   [parameter()][alias("d")][string]$DAYS,
   [parameter()][alias("p")][string]$DIRPATH,
   [parameter()][alias("r")][switch]$RECURSIVE,
   [parameter()][alias("h")][switch]$HELP
)
$O="$HOME\.odrive\common"
$UNSYNC_BIN="$o\odrive.exe"
if ($RECURSIVE) {
   $PARAM = @{Recurse = $true}
}
else {
   $PARAM = @{Recurse = $false}
}
if ($HELP) {
   echo "Usage: unsync_by [-e <extension>] [-s <size in kilobytes>] [-d <days>] [-p <directory path>] [-r]"
   echo "Help: Unsync files by extension, size, or days old for a given directory"
   echo "Options:"
   echo "-e -extension Unsyncs files with the specified extension"
   echo "-s -size Unsyncs files larger than the specified size in kilobytes"
   echo "-d -days Unsyncs files older than the specified day"
   echo "-p -dirpath The specified path"
   echo "-r -recursive Unsyncs files recursively through the specified path"
   echo "-h -help Help"
   break
}
if(!(Test-Path -Path $UNSYNC_BIN)) {
    echo "Downloading CLI binary ... "
    (New-Object System.Net.WebClient).DownloadFile("https://dl.odrive.com/odrivecli-win", "$O\oc.zip")
    $shl=new-object -com shell.application
    $shl.namespace("$O").copyhere($shl.namespace("$O\oc.zip").items(),0x10)
    del "$O\oc.zip"
    echo "Done!"
}
if (-Not ($DIRPATH)){
   echo "Invalid arguments. Please consult help for usage details (-h, -help)." 
   break
}
elseif ($EXTENSION) {
   echo "unsyncing all files of $EXTENSION type in $DIRPATH"
   Get-ChildItem -File -Path "$DIRPATH\*" -Filter "*.$EXTENSION" @PARAM | % {& "$UNSYNC_BIN" unsync "$($_.FullName)"}
}
elseif ($SIZE) {
   echo "unsyncing all files larger than $SIZE kilobytes in $DIRPATH"
   Get-ChildItem -File -Path "$DIRPATH\*" -Exclude *.cloud* @PARAM | Where-Object {$($_.Length) -gt "$($SIZE)KB"} | % {& "$UNSYNC_BIN" unsync "$($_.FullName)"}
}
elseif ($DAYS) {
   echo "unsyncing all files older than $DAYS days in $DIRPATH"
   $SUBDATE = $((Get-Date).AddDays(-$($DAYS)))
   echo $SUBDATE
   Get-ChildItem -File -Path "$DIRPATH\*" -Exclude *.cloud* @PARAM | Where-Object {$($_.LastWriteTime) -lt $SUBDATE} | % {& "$UNSYNC_BIN" unsync "$($_.FullName)"}
}
else {
   echo "Invalid arguments. Please consult help for usage details (-h, -help)."
}

**You may have noticed that I am focusing on shell scripting in this post. It has proven to be the most generally accessible method for our users. However, if you are proficient in Python, please feel free to add code directly in the odrive.py client file that we supply. This adds even more possibilities for expansion**

You’ve taken your first step into a larger world

Whew! I know this may have been a bit of a bumpy ride if it you are still new to CLIs, but I hope that you were able to come away with a better understanding of just how powerful they can be, especially when sync is involved.

This is just one of the many ways we are trying to transform how users interact with their storage. The sync CLI is available now, for all odrive users.

Don’t worry, you can now pull your poor mouse out of the trash and get back to clicking. Speaking of which, the first thing that you should probably click on is the lovely odrive logo down there. ↓ ;)

Sign up for free!

odrive supports Dropbox, Google Drive, Amazon Drive/Photos, Microsoft OneDrive, Slack, SharePoint, Office 365, Facebook, Microsoft OneDrive for Business, Box, Amazon S3, Wasabi, DigitalOcean Spaces, DreamHost DreamObjects, MinIO, S3 Compatible Storage, Google Cloud Storage, Backblaze B2, FTP, FTPS, SFTP, WebDAV, 4shared, ADrive, HiDrive, and Yandex Disk

For more information about our Linux client, take a look at our documentation.

Have odrive CLI/Agent ideas/questions/requests, or just want to add to the conversation? Hit us up on the odrive forum.

Read More

Sync Differently

Unsync Is the Missing Link to Cloud Storage

Kernel Panic! Dropbox Infinite Is Getting Too Intimate

Avenging The Nerds

Is Your Cloud Storage Slow as Shit?