Whatbox Logo
Login for certain variables to be updated with your slot's information

Wiki > rsync

rsync is a utility for efficiently transferring and synchronizing files between a computer and an external hard drive and across networked computers by comparing the modification times and sizes of files.

Example usage: Changing Whatbox servers

This assumes that you have bought a new slot while you still have access to your first slot. Open a support ticket if you want Whatbox staff to handle your transfer, or use the self-service plan change page.

Before starting the transfer

Shutdown rtorrent on the new slot with pkill -f rtorrent to prevent any torrents from accidentally being added before the transfer is over. Leave the client offline until the data synchronization has completed.

If you have mounted an external filesystem to your slot using a tool such as rclone or plexdrive, be sure to either (a) unmount it before starting the transfer, or (b) exclude the path to the mount in the rsync command below with --exclude '/path/to/mount' (stick this before the src and dest paths). This will prevent rsync from parsing through your mounted files and transferring them.

Transfer data

  1. SSH into your new server.
  2. Create a new screen that rsync will run in. screen -S rsync
  3. Run this command from the new server. Make sure to replace "oldserver" in the command with your previous server's name. Run both commands to copy all of your slot's files: rsync -avhHSP --stats user@oldserver.whatbox.ca:/config/user/ /config/user/ rsync -avhHSP --stats user@oldserver.whatbox.ca:/home/user/ /home/user/
  4. Enter your old slot's password when it asks.
  5. Detach from the screen and let rsync do its business. Ctrl+A, d
  6. Give it some time. If you'd like to reattach to the rsync screen, simply screen -r rsync

The breakdown of the rsync command is as follows
rsync - Runs the syncing program rsync
a - archive mode; equals -rlptgoD (no -H,-A,-X)
v - increase verbosity
h - output numbers in a human-readable format
H - preserve hard links
S - handle sparse files efficiently
P - same as --partial --progress (keep partially transferred files and show progress during transfer)
--stats - give some file-transfer stats

After the transfer is done

You will need to restart your client for it to acknowledge new files and torrents, this can be done by hitting the Restart link on your manage page.

If you're getting 0% complete for all of your torrents, try checking the path where it looks for the files, if it's something like /mnt/sd*1/user/files/, try changing it to /home/user/files/ and rechecking the torrents.

Regularly downloading files with rsync

With a few extras.

Using rsync is by itself pretty straightforward, but you can also get a bit more out of it. Setting this up will download anything new when it is finished, and send you an email listing of what was downloaded since the last email, with usage information from Skynet. Using cron you decide how often it runs, and within the script you decide at what hour you want the mail.

Requirements

For this to work properly you need to be using rTorrent and passwordless SSH authentication. Add the following line to your rtorrent.rc and create a directory called "finished" in the root of your home directory.

method.set_key = event.download.finished,link3,"execute2=ln,-s,$d.base_path=,/home/user/finished"

This makes a symbolic link in ~/finished when whatever you are downloading is completed.

You will also need ncat (from nmap), and msmtp on your local *nix server. ncat is needed to get info from Skynet, and msmtp needs to be configured separately for mail to work. This guide should be good for that.

The script below should be saved to a .sh file, and use cron to run it as often as you like.

Example mail

Downloaded File List:
 Folder.1
 Folder.2
 File1.mkv
 File2.rar


Transfer successful. Deleted following symbolic links:
/home/user/finished/Folder.1
/home/user/finished/Folder.2
/home/user/finished/File1.mkv
/home/user/finished/File2.rar

server.whatbox.ca Status:
server 6.27 GB/100.00 GB
Heat1 on server  1.07 TB / 2.50 TB
server - Heat1 Expires in 1 month, 6 days, 20 hours on Jan 21st, 2013

Local computer disk usage:
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb1       931G  510G  375G  58% /mnt/sdb1

The Script

# This is a script which works for me, based on the previous script submitted here and then further tailored to suit my preferences. Use at your own risk etc.

#!/bin/sh

# Configuration, should be all set up.
remoteuser="user"
remoteserver="server.whatbox.ca"
sourcefolder="/home/user/finished/"
targetfolder="/your/local/folder"
templog=`mktemp -t templog.XXXX`
templogfind=`mktemp -t templogfind.XXXX`
syncvar="-avzPL -e ssh --log-file=$templog --bwlimit=4096"

# Except this part.
mailto="receiving mail address"
mailfrom="outgoing mail address"

tempmail=".tempmail"

# Mail out an error report when called by a general signal trap
    cleanfail () {
    printf "To: $mailto\nFrom: $mailfrom\nSubject: Whatbox transfers - From `date +%d-%m-%Y-%H:%M:%S`\n\nWhatbox synchronisation has failed due to unspecified signal error.\n\n" > $tempmail
    cat $tempmail | msmtp -a default $mailto
    rm $tempmail
    rm $templog
    rm $templogfind
    exit 1
    }
    
# Mail out an error report when called by an rsync trap.
    cleanfail_rsync () {
    printf "To: $mailto\nFrom: $mailfrom\nSubject: Whatbox transfers - From `date +%d-%m-%Y-%H:%M:%S`\n\nWhatbox rsync has failed. Rsync log attached.\n\n" > $tempmail
    cat $templog >> $tempmail
    cat $tempmail | msmtp -a default $mailto
    rm $tempmail
    rm $templog
    rm $templogfind
    exit 1
    }

# Mail out an error report when called by a find trap.
    cleanfail_find () {
    printf "To: $mailto\nFrom: $mailfrom\nSubject: Whatbox transfers - From `date +%d-%m-%Y-%H:%M:%S`\n\nWhatbox symbolic link cleanup has failed. Find process log attached.\n\n" > $tempmail
    cat $templogfind >> $tempmail
    cat $tempmail | msmtp -a default $mailto
    rm $tempmail
    rm $templog
    rm $templogfind
    exit 1
    }

# Mail out an error report when called by an existing instance of rsync already running trap.
    cleanfail_rsync_active () {
    printf "To: $mailto\nFrom: $mailfrom\nSubject: Whatbox transfers - From `date +%d-%m-%Y-%H:%M:%S`\n\nrsync process is still active. Large download may still be in progress.\n\n" > $tempmail
    cat $tempmail | msmtp -a default $mailto
    rm $tempmail
    rm $templog
    rm $templogfind
    exit 1
    }

# Check if rsync is already running. If so, exit with an email notification reporting transfers still in progress.
pgrep rsync >/dev/null 2>&1 && cleanfail_rsync_active

# Touch timestamp file to prevent deleting links that are created while rsync is running
ssh $remoteuser@$remoteserver touch $sourcefolder/.rsync-timestamp

# Trap error signals
trap cleanfail 1 2 3 15

# If all is well so far, commence the rsync automated download from the seedbox. Run it twice in case if more files came in during the first rsync process.
rsync $syncvar $remoteuser@$remoteserver:$sourcefolder $targetfolder
rsync $syncvar $remoteuser@$remoteserver:$sourcefolder $targetfolder

# Trap any errors, log and email them if rsync does not terminate with a successful exit value (0). Otherwise, continue to delete symlinks now that we're done with them. If the deletion of symlinks fails, this error is also trapped, logged and emailed.
if [ $? != 0 ]; then
    cleanfail_rsync
else
    printf "\n\nTransfer successful. Deleted following symbolic links:\n" > $templogfind
    ssh $remoteuser@$remoteserver find $sourcefolder \! -newer $sourcefolder/.rsync-timestamp -type l -delete -print >> $templogfind
    if [ $? != 0 ]; then
        cleanfail_find
    else
    :
    fi
fi

# Mail out report for a successful transfer. List a trimmed down version of rsync log file, trimming down to parent folders and otherwise single files. If you prefer the full rsync log file, just uncomment the line after the following printf and comment out the cut line after it.
printf "To: $mailto\nFrom: $mailfrom\nSubject: Transfers - From `date +%d-%m-%Y-%H:%M:%S`\n\nDownloaded File List:\n" > $tempmail
# cat $templog >> $tempmail
# I believe this cut line is simpler than using awk with print arguments. This series of commands looks for the "+" in the rsync log file and removes everything up to and including the last "+". Then grep filters out the first supposed file in the list which is just "./". After this, sed takes over and strips all the subdirectories from the first "/" encountered after the parent directory name(s). All that's then left to do is to wipe out all the duplicate lines so that all that remains is each parent directory without any duplication and unique files. It essentially creates the same list as what's in the /finished/ folder but at least it's from a confirmed rsync log file rather than working on assumptions, just to keep the script robust.
cut -s -d + -f 8 $templog | grep -v "\./" | sed 's%/.*%%' | uniq >> $tempmail

# Add the symlink deletion results
cat $templogfind >> $tempmail

# Collect usage stats from Skynet IRC bot and add to email report
printf "\n$remoteserver Status:\n" >> $tempmail
(
    echo NICK $remoteuser
    echo USER $remoteuser 8 user : $remoteuser
    sleep 3
    echo "PRIVMSG Skynet" !usage
    sleep 1
    echo "PRIVMSG Skynet" !traffic
    sleep 1
    echo "PRIVMSG Skynet" !expires
    sleep 1
    echo QUIT
) | ncat --ssl irc.whatbox.ca 6697 | grep -w "Skynet!Skynet" | grep -v "Checking...please" | awk -F":" '{ print $3 $4 $5 }' >> $tempmail

# Add local disk stats to email report
printf "\n`hostname`/mnt/sdb1 usage:\n" >> $tempmail; df -h | grep Filesystem >> $tempmail && df -h | grep sdb1 >> $tempmail

# Email the compiled report    
cat $tempmail | msmtp -a default $mailto

rm $templog
rm $templogfind
rm $tempmail
exit 0