Syncing data between two machines
Using Rsync to sync directories on a single machine
Creating Remote Tar Archives
 -Notes

---

Remote backups to tape, or to a file are actually pretty easy to do in Unix/Linux. O'Reiley publishes a book called "Linux server hacks" that contains similar examples.

Syncing data between two machines

• Secure copy can be used to copy individual files or a whole directory over SSH. The following command copies the contents of "data" to the local working directory. The disadvantage is that it doesn't preserve ownership info, symbolic links, and isn't recursive (big disadvantage). Tar is better for this stiuation. Note that this works on large files in Unix/Linux, but fails with Cygwin (fails around 699 Megs).

scp -C user@machine.ocean.dal.ca:/home/user/data/* .

• Rsync is also a very good way to sync two machines. Running it over SSH is a convenient and easy way of doing it securely. Even if an original file is deleted, rsync will retain it on the backup. To sync all the contents (and subdirectories) of /home/user/stuff on machine.ocean.dal.ca to machine2.ocean.dal.ca, run the following command from machine2:

rsync -ave ssh user@machine.ocean.dal.ca:/home/user/stuff .

(this one also prints stats to the screen, and uses compression):

rsync --stats --progress -avze ssh user@machine.ocean.dal.ca:/home/user/stuff .

• An easy rotating backup using rsync and hardlinks. Rsync will sync the remote directory to the current directory. The cp command (with -alf) will copy the current/dir directory to another directory dictated by the day (Monday, Tuesday, etc...). The -l option means that all files will be copied as hardlinks, which means it won't take any extra space on the disk. If the file under current changes or is deleted, the hardlink will stay unchanged (whereas a symbolic link always points to the source file or directory). This approach will create a daily backup that will use slightly more space than a single backup.

#!/bin/bash
DAY=`date +%A`
rsync -ave ssh --delete user@machine.phys.ocean.dal.ca:/home/user/path \
 /home/local_user/path/current/
cp -a -l -f /home/local_user/path/current/dir /home/local_user/path/$DAY

• A cron job can be set up on the local machine to automate this backup. Creating a text file (eg: cron_job.cron) containing the following line will execute this script (eg: the previous rsync script) every day at 3 am in the morning. Add this to the crontab by typing the command crontab cronjob.cron at the prompt.

0 3 * * * /home/user/cron_jobs/backup_script.sh

• The use of ssh keys will eliminate the need for passwords, while keeping a secure connection.

Using Rsync to sync directories on a single machine

• Perhaps you have a complicated directory structure on a computer that you would like to sync to another directory without copying the whole thing every time. Rsync can be used to do this while only copying the files that have changed:

rsync -av /home/user/bigDir /home/user/newdir

The previous command will recursively copy bigDir as a new subdirectory of newdir. This command is only slightly more complicated than the cp command.

Creating Remote Tar Archives

• The following command will create a compressed .tar.gz archive on your local machine (foo.tar.gz) from the ~/junk/seistrans_test directory on a remote machine (in this case fox.ocean.dal.ca). Once the login password is supplied, it will proceed as if it was a local archive.

ssh fox.ocean.dal.ca "tar -cvf - junk/seistrans_test" > foo.tar

• If a backup of the entire machine is desired, execute the following:

 ssh fox.ocean.dal.ca "cd /; tar -cvf - *" > foo.tar 

This example fetches a file from a remote server, compresses it on the local machine and saves the output:

ssh user@machine.ocean.dal.ca "cat /dir/file.sgy" | bzip2 --stdout > file.sgy.bz2

• This example shows how to put the data on the remote machine from the local machine. It directs the local xyz2clar directory (and all of its contents) to the file ~/foo2.tar.gz on Fox.

tar -cvf - xyz2clar/ | ssh fox.ocean.dal.ca "cat > foo2.tar"

• or use gzip compression to create a smaller archive:

tar -zcvf - xyz2clar/ | ssh fox.ocean.dal.ca "cat > foo2.tar.gz"

• These commands can be combined with the tape drive (once the directions listed in VxaTapeDrive are followed). Please refer to VxaTapeDrive for detailed instructions.

• For example: A large (80 gig) tape has been placed in the VXA tape drive on Fox, and is ready to backup. Perform the following command from Fox (local machine), to backup the entire contents of machine.ocean.dal.ca (remote Unix/Linux machine):

mt -f /dev/st0 setblk 0
ssh user@machine.ocean.dal.ca "cd /; tar -cvf - *" > /dev/st0

• Restoring this backup would be quite similar:

mt -f /dev/st0 setblk 0
ssh user@machine.ocean.dal.ca "cd /; tar -xvf -" < /dev/st0

• NOTE: The remote machine must have SSH running. If it doesn't (ie: it has a new hard drive and you wish to restore from backup), Knoppix »http://www.knoppix.net/ is an easy way to get an SSH server running on a bare machine.

Notes

• If tar backs up a file of 0 bytes (these are quite common) it will say tar: Error exit delayed from previous errors, once the archive is complete. This error doesn't seem to matter since everything gets backed up. A file containing directories/files to exclude from the archive can be given as an argument to the -X option.