Linux Backup Script(s)

using 'rsync'

home directory backup - sample code

Home > RefInfo menu > Computer topics menu > This 'rsync' Backup Scripts page

Note: The scripts here may be touched up or revised occasionally.

INTRODUCTION : (2010 Apr)

In early 2010, I happened to read about using the Linux 'rsync' command to backup files. Reference: Linux Format Magazine #130, Apr 2010, page105, in the Answers section.

I devised the following scripts, for my main desktop computer, based on that information. I run the 'rsync' backup script whenever I want to back up my home directory (to an external USB disk drive).

    See the comments (below), in the backup script(s), for a description of the 'rsync' parameters being used --- such as '--archive' and '--delete'.

In October 2010, when I fried my video board (and, as I later found out, my CPU) on my main desktop computer when investigating a noisy fan, my 'rsync' backup really saved me. I was able to restore about 45 Gig of backed up data to some of my other computers --- which I used until my desktop CPU and video were replaced/repaired.

I only wish I had backed up more frequently. I have now made a backup rsync-script an icon on my desktop and I run the script just before I logoff --- especially after login sessions in which I change some of the files in my web site directories and software development directories.


Image of the 'rsync' backup script icon.

Having the backup script as an icon on my desktop --- near a logoff icon --- makes it very easy for me to remember to 'kick off' the backup script.

    (I also run a file-cleanup script before running the 'rsync' script. The 'fileCleanup' script is described below.)

    (Lousy EVGA Nvidia 7600GS board! Googling revealed that many others have had problems with a noisy fan on this board --- sometimes only a few months after installing the board. Many reported burst capacitors on this board --- and the board kept running, until one too many capacitors blew. My board lasted for about 3 years --- progressively noisier for about the last year of those three.

    It appears that the screws holding a fan onto the board were working their way loose, and the fan was touching the screws. The noise got so bad I could not ignore it anymore. I am a fan of fanless video boards now --- pun intended. I do not need a fan-equipped, high-end video board, since I am not a video game fan --- pun intended.)

Script for backing up home directory, with 'rsync' :   (2010 Apr version)

File is not shown. Technical difficulties.

UPDATE:   (2010 Dec version of the script above)

    This new version allows for starting up 'rsync' from a desktop icon. This version starts up 'rsync' in a terminal window --- using the 'gnome-terminal' command --- so that backup messages can be seen in the terminal. This helps assure that the proper (recently changed) files are being backed up, and one can see when the backup is finished (when the messages stop and a summary is shown).

    This script uses a little 'start_sh_after_exe.sh' script whose code is shown further below on this page. For command-syntax details, see comments and code in this script.

Script for backing up home directory, with 'rsync' :   (2010 Dec version)

File is not shown. Technical difficulties.

PREPARING AN EXTERNAL BACKUP DRIVE : (2010 Oct)

As the backup directory name (/media/Maxtor160GB) in the rsync script above indicates, the backup drive on that computer (a home-assembled PC) was an external Maxtor 160 Gigabyte drive.

In October 2010, I prepared a 1 Terabyte Seagate Freeagent external drive for use as a backup drive --- to be used with an 'rsync' script on another desktop computer of mine --- a Cyberpower PC that I bought in 2005 from cyberpowerpc.com.

After frying the video board on my main desktop computer, I installed Ubuntu 9.10 on the Cyberpower computer in October 2010 --- overlaying an old Xandros Linux installation, as I described in a web page on installing Ubuntu 9.10 over Xandros Linux, while preserving a dual-boot of MS Windows --- and creating a separate partition for the home directory.

I took this opportunity to take notes on how I prepared this Seagate drive as a backup drive for a Linux system.

A description of the drive preparation process, for this 1 Terabyte Seagate drive, follows.


  1. I plugged in the USB and power cords of the disk drive.

  2. A 'Freeagent Drive' icon showed up on my Ubuntu desktop.

  3. I right-clicked the icon and chose 'Open'. I saw a bunch of Seagate files (for MS Windows Seagate software) --- about 0.5 Gig of directories and files. To backup those files, in case I ever had use for them, I copied those files to my home directory --- into new directory 'DISKseagateFreeagent'.

  4. Then I right-clicked the Seagate drive icon and chose 'Format...'. In the Format GUI, I chose an 'ext3' file system in the 'Type:' drop down menu, and I specified 'SEAGATEfreeagent1000G' at the 'Name:' entry field.

    I clicked on the 'Format' button. It took about 12 to 15 minutes to format the 1 Terabyte drive. When the formatting was done, I right-clicked the drive icon and checked 'Properties'. The volume name was shortened to 'SEAGATEfreeagent' --- 16 characters. Apparently, that is the maximum length of the volume name.

    The 'Properties' showed that about 850 Gig was available after formatting the 1 Terabyte drive. (There is about 150 Gig of overhead for the ext3 file system??)

  5. I prepared an rsync script in $HOME/Rsync on the Cyberpower PC --- script name: 00_rsync_bkup_2freeagentSEAGATE.sh

    The one command in that script is (like the script above):

           gnome-terminal -e \
           rsync --archive --delete -vv --modify-window=1 \
                 /home/blaze/ /media/SEAGATEfreeagent/CyberpowerPC/home/blaze/
    
           
  6. I ran the script. I got 'file does not exist' errors. I had to make the 'to' subdirectories --- CyberpowerPC/home/blaze/ --- on the Seagate drive.

    I right-clicked the Seagate drive icon and chose 'Open'. In the Nautilus file manager that opened, I made a 'CyberpowerPC' directory with 'File > Create Folder'. I opened that directory and made a 'home' directory with 'File > Create Folder'. I opened that directory and made a 'blaze' (username) directory with 'File > Create Folder'.

  7. I ran the script again --- in a terminal window positioned in the directory where I had the script --- $HOME/Rsync.

    The names of files being archived for the first time streamed down the terminal window. There are about 45 Gig of files in the home directory. The copying went on for about 35 minutes. That's a rate of about 45/35 = 1.3 Gig per minute.

  8. I changed the parameter '-vv' to '-v' to speed up the running of the script. '-vv' shows a list of all files being processed, which is many thousands in my 45 GB of files. '-v' just shows the files being deleted and the files being backed up, in the backup directory.

    I am now prepared to run this rsync script almost every day, at the end of a long file-generating and/or file-modifying login session.

    I made a desktop icon for the script and positioned the icon under the Shutdown drop down menu at the right end of the top Gnome panel.

    The 'gnome-terminal -e' part of the command above is intended to cause a terminal window to open, in which I can watch the names of files being archived --- so that I can follow the progress of the backup --- and shutdown the computer when it is done.

    Since I generally have less than 1 Gig of file changes at the end of a login session, I expect the rsync script to run for less than a minute, in most cases. (It turns out it usually runs for a little more than one minute, since it still has to check all 45 Gig of files for differences.)


Actually, the 'gnome-terminal -e' command in step 5 above is a simplification of what I actually use.

I would like the gnome-terminal to stay open after the rsync command executes, so that I can examine the list of files being archived. But 'gnome-terminal' does not support a '-hold' option like 'xterm' does.

I could use 'xterm -hold -bg black -fg white -e'. But, since I prefer the scroll bar of 'gnome-terminal', I use the following command instead.


  gnome-terminal -e "$HOME/Rsync/start_sh_after_exe.sh \
      rsync --archive --delete -v --modify-window=1 \
         /home/blaze/ /media/SEAGATEfreeagent/CyberpowerPC/home/blaze/"

where the script 'start_sh_after_exe.sh' contains the following code.

#!/bin/sh
set -x
$@
set -
/bin/bash

I found a forum thread on the internet where someone said they use this technique to cause the 'gnome-terminal' to stay open after executing the command string. (In fact, not only does it stay open, but it can accept additional commands at the command prompt.) (I added the 'set -x' and 'set -' commands --- to show the command string being executed --- the '$@'.)


Here is the complete image of the 'rsync' script that I use with a Seagate drive --- complete with the comments that I use to help remind myself of some things about doing backup with the 'rsync' command.

The code (with comments) for the 'start_sh_after_exe.sh' script is shown further down on this page.

File is not shown. Technical difficulties.

FILE CLEANUP BEFORE BACKUP :
(A 2010 Oct UPDATE on the 'rsync' backup process before logoff)

After doing several backups just before logoff (via the 'backup_2maxtor' desktop icon shown above), I saw, in the messages from 'rsync', that 'rsync' was repeatedly backing up files from browser 'cache' directories --- and thumbnail image files from a $HOME/.thumbnails directory.

This was a waste. I did not need these files/subdirectories backed up.

    (I have not traced the source of all of the thumbnail files. I have set my Nautilus preferences to NOT show thumbnail icons of image files. However, Nautilus seems to continue making thumbnails, in some cases. The Gnome2 'SaveAs...' dialog, which includes a GUI directories-and-files navigator, seems to be making some of the thumbnail files. It also appears that some image programs, such as the 'Eye of Gnome' ('eog') image viewer program, makes some of the thumbnail files.)

I devised the following 'files-cleanup' script to show me the removal of files from such directories.

2010-2011 Version for 'file-cleanup':

File is not shown. Technical difficulties.


UPDATE - 2012 Sep Version of 'file-cleanup':

    (This version uses just one terminal window, instead of one terminal for each of several directories to be cleaned. Since the '-e' option of this terminal command is not friendly to a lot of separate commands, the terminal-startup command calls on a separate script, named '00_fileCleanup_ver2_RunInTerminal.sh', to do the file cleanup.

    For command-syntax details, see the comments and code in the following two scripts.)

SCRIPT '00_fileCleanup_ver2.sh'

File is not shown. Technical difficulties.

SCRIPT '00_fileCleanup_ver2_RunInTerminal.sh'

File is not shown. Technical difficulties.

Of course, if I find that I do not need to clean one of these directories, I can comment-out that cleanup code. And I could add other directories to cleanup.

    I hardly ever use Firefox (I use Seamonkey instead), so in a perfect world I would not need to clean out the Firefox cache directories. But there are some applications which startup Firefox (to show help files or whatever), so I keep the Firefox cache directory cleanup code even though it usually finds nothing to delete.

    You might want to tell me that one can clean the Firefox cache directory by setting cache-cleanup-at-exit in Firefox preferences. But I find that the Firefox (and Seamonkey) cleanup is not always complete.

    The reason for the two Seamonkey directories: I never know when a version of Seamonkey is going to use the $HOME/.seamonkey directory or the $HOME/.mozilla/.seamonkey directory. There has been some confusion in use of these directories in a previous Seamonkey 1.x release. (The $HOME/.seamonkey directory does not seem to be used by Seamonkey2.)

Like in my 'rsync' backup script(s), I use a call to the following 'start_sh_after_exe.sh' script, from 'gnome-terminal -e', to allow for holding the gnome-terminal open after each 'rm' command runs --- so I can verify the command(s) being run.

File is not shown. Technical difficulties.


To help remind myself to (optionally) run the file cleanup script before running the rsync backup script (via the desktop icon), I added an icon to the desktop (above the rsync backup icon) to remind myself to do the file cleanup before the backup. Picture follows.

Sometimes I run the 'file-cleanup' script without running the 'backup' script, just before logging off. Hence I keep the two scripts separate, rather than combining them into one 'monolithic' script.

BACKUP DISK FOR A NETBOOK :
(a WHOOPS moment --- a few panicky minutes, 2010dec02 ---
use of 'fdisk' and 'Gparted' to avoid a 'bricked' USB drive)

After setting up external backup disks for a couple of desktop computers, I decided to set up an external backup disk for an Acer netbook computer on which I installed Ubuntu Linux 9.04 in mid 2009, as described on my Ubuntu Installs web page. I bought a Segate Goflex 320BG portable hard drive at BestBuy.

I connected the drive to a USB port of my netbook and it was automatically recognized by the Ubuntu Linux installation on my netbook. A drive icon appeared on the desktop. I right-click opened the drive to check the contents.

There are about half a Gig of files on most Seagate drives --- mostly Microsoft Windows executables, which I did not need on my Acer netbook on which I had installed Ubuntu Linux 9.04 over the MS Windows OS that came on the netbook.

    (By the way, those articles that keep appearing that say Linux is dead on netbooks --- I strongly disagree. I doubt if I am the only one who buys a netbook with Microsoft Windows on it and overlays that MS installation with a Linux install. I now have 3 Acer netbooks on which I overlaid MS Windows installations with Ubuntu Linux installations.)

I moved the Seagate files into a new home subdirectory, just in case there were a file (like a documentation file) that I might need later.

The drive label was a rather generic name like 'Seagate', but I like to use a more descriptive label, like 'SeagateGoflex320GB' that shows the capacity of the drive. So I 'right-clicked' on the drive icon and clicked on the 'Properties' line in the menu that popped up.

I was concerned that the Format option was grayed out. I did not know how I would format the drive from NTFS to a Linux format. But I proceeded to try to change the label. I entered a new name for the drive in a blank entry field that showed up. I unmounted the drive and then tried reconnecting it. WHOOPS! I got the following error in a popup window:

    Cannot mount volume.
    Error org.freedesktop.Hal.Device.UnknownError
    libhal.c 1399: wrong reply from hald.
    Expecting an array.
    org.freedesktop.Hal.Device.Volume.UnknownFailure.

Apparently the Hal (Hardware Abstraction Layer) daemon did not like what Nautilus (the Gnome2 file manager) and I did to the drive. I had a mild panic attack. It seemed that I had made a paper-weight out of this new disk drive.

I Googled keywords like 'usb external hard drive ubuntu'.

On an Ubuntu help page on USB mounts, I found a hint: use 'sudo fsdisk -l' to show the disks. With this command (in a terminal window on my Acer netbook), I saw that /dev/sdb1 was recognized as being a 320GB drive --- in addition to /dev/sda, the 160GB internal drive. The fact that the drive could be detected gave me hope that this mount problem could be overcome.

I saw a web page that suggested using Gparted to fix the label. I used the Ubuntu Software Center to install Gparted on the Acer netbook. Then I used Gparted (which recognized the existence of the /dev/sdb1 device) to reformat the NTFS-formatted external drive to ext3. I set the label to 'SeagateGoflex320'. (I entered 'SeagateGoflex320GB' but, apparently, only the first 16 characters were used.) Then I was able to mount the external drive by unplugging the USB connection and reconnecting. I felt much relieved at this point. I could use the drive for something other than a paper-weight.

I then used the sample scripts above to setup a 'fileCleanup' icon and a 'bkup_2Seagate320GB' icon on my desktop, just below the button on the right of the upper Ubuntu/Gnome panel. Now, before I click on that LockScreen/Suspend/Hibernate/Restart/Shutdown button to shutdown the netbook, if I have edited some files in my home directory, I click on the 'fileCleanup' icon to cleanup files before the backup. Then I connect the backup drive and I use the 'bkup' icon to rsync-backup my files.

Works like a charm --- after that initial mount-problem scare. Thank you Google developers, Ubuntu developers and users, fdisk developers, and Gparted developers. In addition to being thankful for my family, I felt I had to add these thank-you's in the 2010 Thanksgiving season in which this near-fiasco occurred.

Bottom of the Linux Backup Script(s) - using 'rsync' page.

To return to a previously visited web page location, click on the
Back button of your web browser a sufficient number of times.
OR, use the History-list option of your web browser.
OR ...

< Go to start of Code samples, above. >
< Go to Top of Page, above. >
< Go to Computer Topics Menu page. >

Page created 2010 Oct 14. Changed 2014 Dec 09.