Linux Shell Scripts
for
file backups using 'rsync'

SAMPLE CODE for:
- home directory backup
- file cleanup before backup
- listing files in some directories
- copying files from 1 backup drive to another

Home > RefInfo menu >

Computer topics menu (Linux section) >

This 'rsync' Backup Scripts (circa 2019-plus) page

! Note !
The scripts here may be touched up or
revised occasionally, if/when I re-visit this page.

Sections Below:

INTRO     RSYNC-SCRIPT(S)     CLEANUP-SCRIPT(S)

LIST-FILES-SCRIPT(S)     DRIVEtoDRIVE-COPY-SCRIPTS     INFO-SEARCHES

A-BACKUP-DRIVE-PREP-CASE     A-BACKUP-DRIVE-RECOVERY-CASE

BOTTOM-of-PAGE

INTRODUCTION :

Around 2010, I migrated my files (email, web browser bookmarks, documents, photos, etc.) from a Microsoft Windows personal computer to a home-built computer using Linux (Ubuntu 9.10, 'Karmic Koala').

Around that time, I happened to read about using the Linux 'rsync' command to backup files.

    (Reference:
    Linux Format Magazine #130, Apr 2010, page 105, in the Answers section.
    Although that article may not be available on the internet, you could try a web search for it or for similar articles.)

That 'Answer' showed that the form of the 'rsync' command for backing up files was rather simple:

  rsync   --archive   --delete   /path/to/source/   /path/to/dest/

For my applicaton, the 'source' directory is my home directory --- and the 'destination' directory is a directory on an external USB storage drive.

    NOTE: You need the slash at the end of the source and destination directory names if they already exist. If you leave off the slash, rsync tries to make a new sub-directory with that last sub-directory name.

In my case, I could simply use

    /home/

as the entire 'source' directory name --- which would backup my home directory and any other home directory under the '/home' directory.

And (in the Ubuntu operating system), I could use for the 'destination' directory a name like

    /media/userid/VolumeName/home/

The '--delete' option removes files not present under the 'source' directory --- that is, it removes files from the 'destination' directory that are no longer in the 'source' directory.

The '--archive' option causes all file permissions and timestamps to be copied.

If you start with an empty 'destination' directory (say, on an external USB storage drive), the first run of the 'rsync' command would copy ALL the files from 'source' to 'destination'.

In subsequent runs, this form of the 'rsync' command copies only files that are different.

With large files that have changed, it only copies the parts that have been altered.

So this form of the 'rsync' command is said to do 'incremental backups' --- copying only new files and files that have been changed, to the 'destination' directory --- and deleting files from the 'destination' directory that have been removed from the 'source' directory.

    NOTE:
    I got burned by the '--delete' option once. I accidentally deleted some files from my home directory (by accidentally highlighting too many files in the GUI file manager, when doing a multi-file delete operation). When I ran the 'rsync' commmand it also deleted the files from the 'destination' (backup) directory. So you have to be careful with deletes --- and perhaps keep another backup on a different backup schedule --- say every few months.


Scripts used with the rsync-scripts:
('cleanup' & 'list-files' scripts)

Back in 2010, I devised 'rsync' backup scripts (early versions of the ones presented below) for my main desktop computer --- a 'home-built' computer. Those scripts were based on the 'Linux Format Magazine' information.

For almost 10 years, I used that 'home-built' computer --- with the Ubuntu 9.10 operating system --- and with a few backup-script changes to accomodate some things I learned from using 'rsync' on my home directory.

For example:

In my first few runs of 'rsync' on my home directory, I found that various 'apps' in the Ubuntu operating system were making 'thumbnail' image files in some 'thumbnails' sub-directories --- hundreds of little files --- un-needed and un-wanted by me.

And there were other un-needed-and-un-wanted files accumulating in web browser directories.

I found that, before doing an 'rsync' run on my home directory, I wanted to remove files from directories like the following --- using the 'rm' command in a script.

    $HOME/.cache/thumbnails/normal/
    $HOME/.cache/thumbnails/large/
    $HOME/.cache/thumbnails/fail/gnome-thumbnail-factory/
    $HOME/.cache/thumbnails/fail/mate-thumbnail-factory/

    $HOME/.mozilla/seamonkey/Crash Reports/pending/
    $HOME/.mozilla/seamonkey/eynie5lc.default/minidumps/
    $HOME/.cache/mozilla/seamonkey/eynie5lc.default/ cache2/entries/

    $HOME/.mozilla/firefox/Crash Reports/pending/
    $HOME/.mozilla/firefox/7kxtmyvv.default/minidumps/
    $HOME/.cache/mozilla/firefox/7kxtmyvv.default/ cache2/entries/

    $HOME/.macromedia/Flash_Player/macromedia.com/ support/flashplayer/sys/

    NOTE: (on thumbnail files)
    I was not able to trace the source of ALL of the thumbnail files. When I was using the Gnome2-Nautilus file manager, I set my Nautilus preferences to NOT show thumbnail icons of image files.

    However, Gnome and/or Nautilus seemed to continue making thumbnails, in some cases. The Gnome2 'SaveAs...' dialog, which includes a GUI directories-and-files navigator, seemed to make some of the thumbnail files. It also appears that some image programs, such as the 'Eye of Gnome' ('eog') image viewer program, makes some of the thumbnail files.

I also found that there were 'not-needed' files ('Crash Reports', 'minidumps', 'cache'-files) accumulating in web browser directories (seen above). The files were coming from the web browser that I prefer to use --- Seamonkey. And the same kinds of files appeared in Firefox directories.

    I hardly ever use Firefox (I use Seamonkey instead), so in a perfect world I would not need to clean out the Firefox cache directories. But there are some applications which startup Firefox (to show help files or whatever), so I keep the Firefox cache directory cleanup code even though it usually finds nothing to delete.

    One might expect that one can clean the Firefox cache directory by setting cache-cleanup-at-exit in Firefox preferences. But I find that the Firefox (and Seamonkey) cleanup is not as complete as I would like.

To avoid constantly backing up un-needed 'thumbnail' and web browser files, I created a 'file cleanup script' to be run before doing an 'rsync' run to backup files that were changed or created.

I also made a 'list-directory-files script' to be run at any time --- such as after a run of the 'file cleanup' script or after a run of the 'rsync' script.

The purpose of that 'list-directory-files script' was two-fold:

  • check for files in the 'cleanup' directories, like those listed above

  • check for files accumulating in other directories, especially web browser directories, to see if I should add those directories to the 'cleanup' script.


New versions of my rsync-scripts:

Around 2019, as the disk-drive on that 'home-built' computer was making bad noises, I finally migrated my files (email, web browser bookmarks, documents, photos, etc.) from Ubunutu 9.10 on that 10-year-old computer to an Acer desktop computer running the Ubuntu-MATE 18.04 operating system.

    That installation of Ubuntu-MATE 18.04 and migration of files will be documented on another web page on this site.

In the process of migrating data to the new desktop computer, I rewrote the scripts for

  • 'rync'
  • 'file cleanup'
  • 'listing-some-directory-files'

In this re-write of backup scripts, I made two scripts for each of these functions --- a 'core' script and a 'run-core-script-in-a-terminal' wrapper script.

The main reason for the 'wrapper' script was because I wanted to be able to quickly and easily run these core scripts by clicking on a desktop icon --- like the icons shown in the following image.

I wanted to see the text output from the scripts in a pop-up terminal window.

Each 'wrapper' script simply runs the corresponding 'core' script in a terminal window using a command like

    mate-terminal   -t "title-goes-here"   -e core-script-name

For another operating system environment (other than MATE), one could use another terminal-startup command, such as 'gnome-terminal' or 'xterm'.

When I want to run one of the icons' 'wrapper' scripts, I click on its icon and the terminal window opens and I can see the output coming from the 'rsync' or 'rm' or 'ls' commands.

And I can put 'read' commands in the 'core' scripts to have the scripts stop at certain points --- like after a specific 'rm' or 'ls' command --- in order to be able to examine the output carefully, before it is scrolled out of sight.


When to run the scripts:

When I am doing a lot of web-page or shell-script or Tcl/Tk-script development (almost daily), I may run the 'rsync' script almost daily --- to back up the file changes and new files in my home directory to an external USB disk drive.

Otherwise, if I have changed only a few non-critical files, I may run the 'rsync' script after a week or two.

But BEFORE each 'rsync' run, I run the 'file cleanup' script, in order to get rid of an accumulation of un-needed 'thumbnail' files and web browser files, such as 'Crash reports' and 'minidumps'.

    In October 2010, when I fried my video board (and, as I later found out, my CPU) on my main desktop computer when investigating a noisy fan, my 'rsync' backup really saved me. I was able to restore about 45 Gig of backed up data to some of my other computers --- which I used until my desktop CPU and video were replaced/repaired.


How the scripts are presented, below:

The scripts are presented in two ways below:

  • as a link to a text file
    (click on the link and the file shows in a separate browser window)

  • as a scrollable block of text shown in this window
    (implemented by using the HTML 'object' tag)

You can see the comments, in the backup scripts below, for some details of the parameters being used in the 'rsync' and 'rm' and 'ls' and 'read' commands being used in the scripts.

And you can see the exact way that the directories are being specified in those commands.


If you want to download any of these scripts for use, you can right-click on the '.txt' file link and choose an option such as 'Save Link Target As ...' in the popup menu that appears from your web browser.

After downloading one of these scripts, you can remove the '.txt' suffix from the file name and leave the '.sh' suffix.

Remember to change the permissions on the file to make the file 'executable'.

You can use a command like 'chown 755 filename' or, in a GUI file-manager like MATE-Caja, you can right-click on the filename and choose the 'Properties' option from the menu that pops up. Then open the 'Permissions' panel and click the checkbutton titled 'Allow executing file as program'.

'rsync' SCRIPTS:
(to a Seagate 1TB drive)

CORE script:
01_rsync_home_AcerDesktop1_to_ Seagate1TB_ver1_core.sh.txt

WRAPPER script:
01_rsync_home_AcerDesktop1_to_ Seagate1TB_ver1_RunCoreInTerminal.sh.txt

Alternative display of these two scripts:

CORE: Link to script

WRAPPER: Link to script


As a 'backup-to-the-backup':

I may periodically want to backup my home directories to another external USB drive.

Here are a pair of scripts to run 'rsync' of the '/home' directory to a Seagate 3TB drive mounted on the /media directory.

'rsync' SCRIPTS:
(to a Seagate 3TB drive)

CORE script:
02_rsync_home_AcerDesktop1_to_ Seagate3TB_ver1_core.sh.txt

WRAPPER script:
02_rsync_home_AcerDesktop1_to_ Seagate3TB_ver1_RunCoreInTerminal.sh.txt

Alternative display of these two scripts:

CORE: Link to script

WRAPPER: Link to script

FILE CLEANUP SCRIPTS:
(to use before an 'rsync' run)

CORE script:
01_fileCleanup_AcerDesktop1_ver1_ core.sh.txt

WRAPPER script:
01_fileCleanup_AcerDesktop1_ver1_ RunCoreInTerminal.sh.txt

Alternative display of these two scripts:

CORE: Link to script

WRAPPER: Link to script

LIST-FILES SCRIPTS:
(to use as needed to search some
'usual suspect' directories for
accumulating files)

CORE script:
01_fileLISTcleanDirs_AcerDesktop1_ver1_ core.sh.txt

WRAPPER script:
01_fileLISTcleanDirs_AcerDesktop1_ver1_ RunCoreInTerminal.sh.txt

Alternative display of these two scripts:

CORE: Link to script

WRAPPER: Link to script

DRIVE-to-DRIVE COPY SCRIPTS:
(to copy a drive to 3 different drives)

(These scripts were to be run at the command line,
rather than by clicking on a desktop icon
for a 'wrapper' script.)

Script1:
00_rsync_maxtor160GB_to_Seagate1TB.sh.txt

Script2:
00_rsync_maxtor160GB_to_Seagate3TB.sh.txt

Script3:
00_rsync_maxtor160GB_to_WD250GB.sh.txt

I used these 3 scripts to copy my home-directory-backup files from a 10-year-old backup drive to several newer drives.

Alternative display of these 3 scripts:

To Seagate1TB: Link to script

To Seagate3TB: Link to script

To WD250GB: Link to script

FOR MORE INFORMATION:

To find more information on using 'rsync' for backups (and for info on related file backup topics), you can try WEB SEARCHES on keywords such as the following.

You can also try Wikipedia pages like the following, and following links on those pages for even more information.

PREPARING AN EXTERNAL BACKUP DRIVE :
(2010 Oct ; in Ubuntu 9.10 (2009 October release),
with the Gnome2-Nautilus file manager)

    NOTE:
    The following info is quite old (from an older 'Rsync Backup Scripts' page), but I preserve the info here to give a 'case study' of formatting an external storage drive to use for file backups.

The backup directory name in my old circa-2010 rsync scripts was '/media/Maxtor160GB'. That name indicates that the backup drive on my old 'home-built-circa-2009' computer (a home-assembled PC) was an external Maxtor 160 Gigabyte drive.

In October 2010, I prepared a 1 Terabyte Seagate Freeagent external drive for use as a backup drive --- to be used with an 'rsync' script on another desktop computer of mine --- a Cyberpower PC that I bought in 2005 from cyberpowerpc.com.

After frying the video board on my main desktop computer, I installed Ubuntu 9.10 on the Cyberpower computer in October 2010 --- overlaying an old Xandros Linux installation, as I described in a web page on installing Ubuntu 9.10 over Xandros Linux, while preserving a dual-boot of MS Windows --- and creating a separate partition for the home directory.

I took this opportunity to take notes on how I prepared this Seagate drive as a backup drive for a Linux system --- in particular, how I formatted the drive with an 'ext3' file system.

A description of the drive preparation process, for this 1 Terabyte Seagate drive, follows.


  1. I plugged in the USB and power cords of the disk drive.

  2. A 'Freeagent Drive' icon showed up on my Ubuntu desktop.

  3. I right-clicked the icon and chose 'Open'. I saw a bunch of Seagate files (for MS Windows Seagate software) --- about 0.5 Gig of directories and files. To backup those files, in case I ever had use for them, I copied those files to my home directory --- into new directory 'DISKseagateFreeagent'.

  4. Then I right-clicked the Seagate drive icon and chose 'Format...'.

      UPDATE NOTE:
      This 'Format...' option may no longer be seen when you right-click on a drive-name or drive-icon. Around 2019, I started using Ubuntu-MATE on my desktop and laptop computers. It uses the MATE desktop environment and the Caja file manager. When you right-click on a drive in MATE, the 'Format...' option does not appear. If you have the 'Applications-Places-System' triplet on your MATE 'top-panel', you can probably use 'System > Preferences > Hardware > Disks' to use the 'gnome-disk-utility' to partition-and-format a drive.

    In the Format GUI, I chose an 'ext3' file system in the 'Type:' drop down menu, and I specified 'SEAGATEfreeagent1000G' at the 'Name:' entry field.

    I clicked on the 'Format' button. It took about 12 to 15 minutes to format the 1 Terabyte drive.

    When the formatting was done, I right-clicked the drive icon and checked 'Properties'. The volume name was shortened to 'SEAGATEfreeagent' --- 16 characters. Apparently, that is the maximum length of the volume name.

    The 'Properties' showed that about 850 Gig was available after formatting the 1 Terabyte drive. (There is about 150 Gig of overhead for the ext3 file system??)

  5. I prepared an rsync script in $HOME/Rsync on the Cyberpower PC --- script name: 00_rsync_bkup_2freeagentSEAGATE.sh

    The one command in that script was:


gnome-terminal -e \
   rsync --archive --delete -vv --modify-window=1 \
   /home/userid/ \
   /media/SEAGATEfreeagent/CyberpowerPC/home/userid/

  1. I ran the script. I got 'file does not exist' errors. I had to make the 'to' subdirectories --- CyberpowerPC/home/userid/ --- on the Seagate drive.

    I right-clicked the Seagate drive icon and chose 'Open'. In the Nautilus file manager that opened, I made a 'CyberpowerPC' directory with 'File > Create Folder'. I opened that directory and made a 'home' directory with 'File > Create Folder'. I opened that directory and made a 'userid' directory with 'File > Create Folder'.

  2. I ran the script again --- in a terminal window positioned in the directory where I had the script --- $HOME/Rsync.

    The names of files being archived for the first time streamed down the terminal window. There were about 45 Gig of files in the home directory. The copying went on for about 35 minutes. That's a rate of about 45/35 = 1.3 Gig per minute.

  3. I changed the parameter '-vv' to '-v' to speed up the running of the script. '-vv' shows a list of all files being processed, which is many thousands in my 45 GB of files. '-v' just shows the files being deleted and the files being backed up, in the backup directory.

    I am now prepared to run this rsync script almost every day, at the end of a long file-generating and/or file-modifying login session.

    I made a desktop icon for the script and positioned the icon under the Shutdown drop down menu at the right end of the top Gnome panel.

    The 'gnome-terminal -e' part of the command above is intended to cause a terminal window to open, in which I can watch the names of files being archived --- so that I can follow the progress of the backup --- and shutdown the computer when it is done.

    Since I generally have less than 1 Gig of file changes at the end of a login session, I expect the rsync script to run for less than a minute, in most cases. (It turns out it usually runs for a little more than one minute, since it still has to check all 45 Gig of files for differences.)


Actually, the 'gnome-terminal -e' command in step 5 above is a simplification of what I actually use.

I would like the gnome-terminal to stay open after the rsync command executes, so that I can examine the list of files being archived. But 'gnome-terminal' does not support a '-hold' option like 'xterm' does.

I could use 'xterm -hold -bg black -fg white -e'. But, since I prefer the scroll bar of 'gnome-terminal', I used the following command instead.


gnome-terminal -e "$HOME/Rsync/start_sh_after_exe.sh \
   rsync --archive --delete -v --modify-window=1 \
   /home/userid/ \
   /media/SEAGATEfreeagent/CyberpowerPC/home/userid/"

The 'start_sh_after_exe.sh' script was used to startup a bash shell after the rsync script (that was passed into this script) finished executing. This was mainly to hold the terminal window open so that I could examine the final messages from 'rsync', including the summary statistics.

I do not use that technique anymore. I use the 'wrapper' and 'core' script pairs, like the ones above --- in which the 'wrapper' script starts up a terminal window and executes the 'core' script in that terminal window.

I think the 'wrapper' and 'core' approach is a little less confusing --- and it allows for additional code to be added to either script --- and it avoids having to deal with the only-one-command limitation of the '-e' option of the terminal startup command.

RECOVERY OF A SEEMINGLY 'BRICKED' BACKUP DISK :
(a WHOOPS moment --- a few panicky minutes, 2010dec02 ---
use of 'fdisk' and 'Gparted' to avoid a 'bricked' USB drive)

    NOTE:
    The following info is quite old (from an older 'Rsync Backup Scripts' page), but I preserve the info here to give a 'case study' of using the 'fdisk' command and 'Gparted' utility to get around a problem with mount-errors with an external file-backups drive.

After setting up external backup disks for a couple of desktop computers, I decided to set up an external backup disk for an Acer netbook computer on which I installed Ubuntu Linux 9.04 in mid 2009, as described on my Ubuntu Installs web page. I bought a Segate Goflex 320BG portable hard drive at BestBuy.

I connected the drive to a USB port of my netbook and it was automatically recognized by the Ubuntu Linux installation on my netbook. A drive icon appeared on the desktop. I right-click opened the drive to check the contents.

There are about half a Gig of files on most Seagate drives --- mostly Microsoft Windows executables, which I did not need on my Acer netbook on which I had installed Ubuntu Linux 9.04 over the MS Windows OS that came on the netbook.

I moved the Seagate files into a new home subdirectory, just in case there were a file (like a documentation file) that I might need later.

The drive label was a rather generic name like 'Seagate', but I like to use a more descriptive label, like 'SeagateGoflex320GB' that shows the capacity of the drive.

So I 'right-clicked' on the drive icon and clicked on the 'Properties' line in the menu that popped up.

I was concerned that the Format option was grayed out. I did not know how I would format the drive from NTFS to a Linux format. But I proceeded to try to change the label.

I entered a new name for the drive in a blank entry field that showed up. I unmounted the drive and then tried reconnecting it. WHOOPS! I got the following error in a popup window:

    Cannot mount volume.
    Error org.freedesktop.Hal.Device.UnknownError
    libhal.c 1399: wrong reply from hald.
    Expecting an array.
    org.freedesktop.Hal.Device.Volume.UnknownFailure.

Apparently the Hal (Hardware Abstraction Layer) daemon did not like what Nautilus (the Gnome2 file manager) and I did to the drive. I had a mild panic attack. It seemed that I had made a paper-weight out of this new disk drive.

I Googled keywords like 'usb external hard drive ubuntu'.

On an Ubuntu help page on USB mounts, I found a hint: use 'sudo fsdisk -l' to show the disks. With this command (in a terminal window on my Acer netbook), I saw that /dev/sdb1 was recognized as being a 320GB drive --- in addition to /dev/sda, the 160GB internal drive. The fact that the drive could be detected gave me hope that this mount problem could be overcome.

I saw a web page that suggested using Gparted to fix the label. I used the Ubuntu Software Center to install Gparted on the Acer netbook. Then I used Gparted (which recognized the existence of the /dev/sdb1 device) to reformat the NTFS-formatted external drive to ext3. I set the label to 'SeagateGoflex320'.

    (I entered 'SeagateGoflex320GB' but, apparently, only the first 16 characters were used.)

Then I was able to mount the external drive by unplugging the USB connection and reconnecting. I felt much relieved at this point. I could use the drive for something other than a paper-weight.

I then used shell-scripting to setup a 'fileCleanup' icon and a 'bkup_2Seagate320GB' icon on my desktop, just below the button on the right of the upper Ubuntu/Gnome2-Nautilus 'top-panel'.

Now I was prepared to do the following.

Before I click on the 'top-panel' button for the LockScreen/Suspend/Hibernate/Restart/Shutdown options, to shutdown the netbook, if I have edited some files in my home directory, I click on the 'fileCleanup' icon to cleanup files before the backup.

Then I connect the backup drive and I use the 'bkup' icon to rsync-backup my files.

Works like a charm --- after that initial mount-problem scare. Thank you Google developers, Ubuntu developers and users, fdisk developers, and Gparted developers. In addition to being thankful for my family, I felt I had to add these thank-you's in the 2010 Thanksgiving season in which this near-fiasco occurred.

Bottom of this
Linux Backup Script - using 'rsync' page.

To return to a previously visited web page location, click on the Back button of your web browser a sufficient number of times.

OR, use the History-list option of your web browser.

OR ...

< Go to start of Code samples, above. >

< Go to Top of Page, above. >

Page history:

Page was created 2021 Jun 20.