Linux Shell Scripts
|
! Note !
The scripts here may be touched up or
revised occasionally, if/when I re-visit this page.
Sections Below:
INTRO
RSYNC-SCRIPT(S)
CLEANUP-SCRIPT(S)
LIST-FILES-SCRIPT(S)
DRIVEtoDRIVE-COPY-SCRIPTS
INFO-SEARCHES
A-BACKUP-DRIVE-PREP-CASE
A-BACKUP-DRIVE-RECOVERY-CASE
BOTTOM-of-PAGE
INTRODUCTION : Around 2010, I migrated my files (email, web browser bookmarks, documents, photos, etc.) from a Microsoft Windows personal computer to a home-built computer using Linux (Ubuntu 9.10, 'Karmic Koala'). Around that time, I happened to read about using the Linux 'rsync' command to backup files.
(Reference: That 'Answer' showed that the form of the 'rsync' command for backing up files was rather simple: rsync --archive --delete /path/to/source/ /path/to/dest/ For my applicaton, the 'source' directory is my home directory --- and the 'destination' directory is a directory on an external USB storage drive. NOTE: You need the slash at the end of the source and destination directory names if they already exist. If you leave off the slash, rsync tries to make a new sub-directory with that last sub-directory name. In my case, I could simply use /home/ as the entire 'source' directory name --- which would backup my home directory and any other home directory under the '/home' directory. And (in the Ubuntu operating system), I could use for the 'destination' directory a name like /media/userid/VolumeName/home/ The '--delete' option removes files not present under the 'source' directory --- that is, it removes files from the 'destination' directory that are no longer in the 'source' directory. The '--archive' option causes all file permissions and timestamps to be copied. If you start with an empty 'destination' directory (say, on an external USB storage drive), the first run of the 'rsync' command would copy ALL the files from 'source' to 'destination'. In subsequent runs, this form of the 'rsync' command copies only files that are different. With large files that have changed, it only copies the parts that have been altered. So this form of the 'rsync' command is said to do 'incremental backups' --- copying only new files and files that have been changed, to the 'destination' directory --- and deleting files from the 'destination' directory that have been removed from the 'source' directory.
NOTE:
Scripts used with the rsync-scripts: Back in 2010, I devised 'rsync' backup scripts (early versions of the ones presented below) for my main desktop computer --- a 'home-built' computer. Those scripts were based on the 'Linux Format Magazine' information. For almost 10 years, I used that 'home-built' computer --- with the Ubuntu 9.10 operating system --- and with a few backup-script changes to accomodate some things I learned from using 'rsync' on my home directory. For example: In my first few runs of 'rsync' on my home directory, I found that various 'apps' in the Ubuntu operating system were making 'thumbnail' image files in some 'thumbnails' sub-directories --- hundreds of little files --- un-needed and un-wanted by me. And there were other un-needed-and-un-wanted files accumulating in web browser directories. I found that, before doing an 'rsync' run on my home directory, I wanted to remove files from directories like the following --- using the 'rm' command in a script.
$HOME/.cache/thumbnails/normal/
NOTE: (on thumbnail files) However, Gnome and/or Nautilus seemed to continue making thumbnails, in some cases. The Gnome2 'SaveAs...' dialog, which includes a GUI directories-and-files navigator, seemed to make some of the thumbnail files. It also appears that some image programs, such as the 'Eye of Gnome' ('eog') image viewer program, makes some of the thumbnail files. I also found that there were 'not-needed' files ('Crash Reports', 'minidumps', 'cache'-files) accumulating in web browser directories (seen above). The files were coming from the web browser that I prefer to use --- Seamonkey. And the same kinds of files appeared in Firefox directories. I hardly ever use Firefox (I use Seamonkey instead), so in a perfect world I would not need to clean out the Firefox cache directories. But there are some applications which startup Firefox (to show help files or whatever), so I keep the Firefox cache directory cleanup code even though it usually finds nothing to delete. One might expect that one can clean the Firefox cache directory by setting cache-cleanup-at-exit in Firefox preferences. But I find that the Firefox (and Seamonkey) cleanup is not as complete as I would like. To avoid constantly backing up un-needed 'thumbnail' and web browser files, I created a 'file cleanup script' to be run before doing an 'rsync' run to backup files that were changed or created. I also made a 'list-directory-files script' to be run at any time --- such as after a run of the 'file cleanup' script or after a run of the 'rsync' script. The purpose of that 'list-directory-files script' was two-fold:
New versions of my rsync-scripts: Around 2019, as the disk-drive on that 'home-built' computer was making bad noises, I finally migrated my files (email, web browser bookmarks, documents, photos, etc.) from Ubunutu 9.10 on that 10-year-old computer to an Acer desktop computer running the Ubuntu-MATE 18.04 operating system. That installation of Ubuntu-MATE 18.04 and migration of files will be documented on another web page on this site. In the process of migrating data to the new desktop computer, I rewrote the scripts for
In this re-write of backup scripts, I made two scripts for each of these functions --- a 'core' script and a 'run-core-script-in-a-terminal' wrapper script. The main reason for the 'wrapper' script was because I wanted to be able to quickly and easily run these core scripts by clicking on a desktop icon --- like the icons shown in the following image.
I wanted to see the text output from the scripts in a pop-up terminal window. Each 'wrapper' script simply runs the corresponding 'core' script in a terminal window using a command like mate-terminal -t "title-goes-here" -e core-script-name For another operating system environment (other than MATE), one could use another terminal-startup command, such as 'gnome-terminal' or 'xterm'. When I want to run one of the icons' 'wrapper' scripts, I click on its icon and the terminal window opens and I can see the output coming from the 'rsync' or 'rm' or 'ls' commands. And I can put 'read' commands in the 'core' scripts to have the scripts stop at certain points --- like after a specific 'rm' or 'ls' command --- in order to be able to examine the output carefully, before it is scrolled out of sight. When to run the scripts: When I am doing a lot of web-page or shell-script or Tcl/Tk-script development (almost daily), I may run the 'rsync' script almost daily --- to back up the file changes and new files in my home directory to an external USB disk drive. Otherwise, if I have changed only a few non-critical files, I may run the 'rsync' script after a week or two. But BEFORE each 'rsync' run, I run the 'file cleanup' script, in order to get rid of an accumulation of un-needed 'thumbnail' files and web browser files, such as 'Crash reports' and 'minidumps'. In October 2010, when I fried my video board (and, as I later found out, my CPU) on my main desktop computer when investigating a noisy fan, my 'rsync' backup really saved me. I was able to restore about 45 Gig of backed up data to some of my other computers --- which I used until my desktop CPU and video were replaced/repaired. How the scripts are presented, below: The scripts are presented in two ways below:
You can see the comments, in the backup scripts below, for some details of the parameters being used in the 'rsync' and 'rm' and 'ls' and 'read' commands being used in the scripts. And you can see the exact way that the directories are being specified in those commands. If you want to download any of these scripts for use, you can right-click on the '.txt' file link and choose an option such as 'Save Link Target As ...' in the popup menu that appears from your web browser. After downloading one of these scripts, you can remove the '.txt' suffix from the file name and leave the '.sh' suffix. Remember to change the permissions on the file to make the file 'executable'. You can use a command like 'chown 755 filename' or, in a GUI file-manager like MATE-Caja, you can right-click on the filename and choose the 'Properties' option from the menu that pops up. Then open the 'Permissions' panel and click the checkbutton titled 'Allow executing file as program'. |
'rsync' SCRIPTS:
CORE script:
WRAPPER script: Alternative display of these two scripts: CORE: WRAPPER: As a 'backup-to-the-backup': I may periodically want to backup my home directories to another external USB drive. Here are a pair of scripts to run 'rsync' of the '/home' directory to a Seagate 3TB drive mounted on the /media directory.
'rsync' SCRIPTS:
CORE script:
WRAPPER script: Alternative display of these two scripts: CORE: WRAPPER: |
FILE CLEANUP SCRIPTS:
CORE script:
WRAPPER script: Alternative display of these two scripts: CORE: WRAPPER: |
LIST-FILES SCRIPTS:
CORE script:
WRAPPER script: Alternative display of these two scripts: CORE: WRAPPER: |
DRIVE-to-DRIVE COPY SCRIPTS:
Script1:
Script2:
Script3: I used these 3 scripts to copy my home-directory-backup files from a 10-year-old backup drive to several newer drives. Alternative display of these 3 scripts: To Seagate1TB: To Seagate3TB: To WD250GB: |
FOR MORE INFORMATION: To find more information on using 'rsync' for backups (and for info on related file backup topics), you can try WEB SEARCHES on keywords such as the following. 'incremental file backup linux ubuntu' 'external usb drive file backup linux ubuntu' 'external usb drive format ext3 ext4 for file backup on linux ubuntu' You can also try Wikipedia pages like the following, and following links on those pages for even more information. Wikipedia page - 'File synchronization' Wikipedia page - 'Comparison of file synchronization software' Wikipedia page - 'Incremental backup' Wikipedia page - 'Backup rotation scheme' Wikipedia page - 'Differential backup' Wikipedia page - 'List of backup software' Wikipedia page - 'Comparison of backup software' Wikipedia page - 'Hard_disk_drive#Market_segments' Wikipedia page - 'USB mass storage device class' |
PREPARING AN EXTERNAL BACKUP DRIVE :
NOTE: The backup directory name in my old circa-2010 rsync scripts was '/media/Maxtor160GB'. That name indicates that the backup drive on my old 'home-built-circa-2009' computer (a home-assembled PC) was an external Maxtor 160 Gigabyte drive. In October 2010, I prepared a 1 Terabyte Seagate Freeagent external drive for use as a backup drive --- to be used with an 'rsync' script on another desktop computer of mine --- a Cyberpower PC that I bought in 2005 from cyberpowerpc.com. After frying the video board on my main desktop computer, I installed Ubuntu 9.10 on the Cyberpower computer in October 2010 --- overlaying an old Xandros Linux installation, as I described in a web page on installing Ubuntu 9.10 over Xandros Linux, while preserving a dual-boot of MS Windows --- and creating a separate partition for the home directory. I took this opportunity to take notes on how I prepared this Seagate drive as a backup drive for a Linux system --- in particular, how I formatted the drive with an 'ext3' file system. A description of the drive preparation process, for this 1 Terabyte Seagate drive, follows.
|
gnome-terminal -e \
rsync --archive --delete -vv --modify-window=1 \
/home/userid/ \
/media/SEAGATEfreeagent/CyberpowerPC/home/userid/
Actually, the 'gnome-terminal -e' command in step 5 above is a simplification of what I actually use. I would like the gnome-terminal to stay open after the rsync command executes, so that I can examine the list of files being archived. But 'gnome-terminal' does not support a '-hold' option like 'xterm' does. I could use 'xterm -hold -bg black -fg white -e'. But, since I prefer the scroll bar of 'gnome-terminal', I used the following command instead. |
gnome-terminal -e "$HOME/Rsync/start_sh_after_exe.sh \
rsync --archive --delete -v --modify-window=1 \
/home/userid/ \
/media/SEAGATEfreeagent/CyberpowerPC/home/userid/"
The 'start_sh_after_exe.sh' script was used to startup a bash shell after the rsync script (that was passed into this script) finished executing. This was mainly to hold the terminal window open so that I could examine the final messages from 'rsync', including the summary statistics. I do not use that technique anymore. I use the 'wrapper' and 'core' script pairs, like the ones above --- in which the 'wrapper' script starts up a terminal window and executes the 'core' script in that terminal window. I think the 'wrapper' and 'core' approach is a little less confusing --- and it allows for additional code to be added to either script --- and it avoids having to deal with the only-one-command limitation of the '-e' option of the terminal startup command. |
RECOVERY OF A SEEMINGLY 'BRICKED' BACKUP DISK :
NOTE: After setting up external backup disks for a couple of desktop computers, I decided to set up an external backup disk for an Acer netbook computer on which I installed Ubuntu Linux 9.04 in mid 2009, as described on my Ubuntu Installs web page. I bought a Segate Goflex 320BG portable hard drive at BestBuy. I connected the drive to a USB port of my netbook and it was automatically recognized by the Ubuntu Linux installation on my netbook. A drive icon appeared on the desktop. I right-click opened the drive to check the contents. There are about half a Gig of files on most Seagate drives --- mostly Microsoft Windows executables, which I did not need on my Acer netbook on which I had installed Ubuntu Linux 9.04 over the MS Windows OS that came on the netbook. I moved the Seagate files into a new home subdirectory, just in case there were a file (like a documentation file) that I might need later. The drive label was a rather generic name like 'Seagate', but I like to use a more descriptive label, like 'SeagateGoflex320GB' that shows the capacity of the drive. So I 'right-clicked' on the drive icon and clicked on the 'Properties' line in the menu that popped up. I was concerned that the Format option was grayed out. I did not know how I would format the drive from NTFS to a Linux format. But I proceeded to try to change the label. I entered a new name for the drive in a blank entry field that showed up. I unmounted the drive and then tried reconnecting it. WHOOPS! I got the following error in a popup window:
Cannot mount volume. Apparently the Hal (Hardware Abstraction Layer) daemon did not like what Nautilus (the Gnome2 file manager) and I did to the drive. I had a mild panic attack. It seemed that I had made a paper-weight out of this new disk drive. I Googled keywords like 'usb external hard drive ubuntu'. On an Ubuntu help page on USB mounts, I found a hint: use 'sudo fsdisk -l' to show the disks. With this command (in a terminal window on my Acer netbook), I saw that /dev/sdb1 was recognized as being a 320GB drive --- in addition to /dev/sda, the 160GB internal drive. The fact that the drive could be detected gave me hope that this mount problem could be overcome. I saw a web page that suggested using Gparted to fix the label. I used the Ubuntu Software Center to install Gparted on the Acer netbook. Then I used Gparted (which recognized the existence of the /dev/sdb1 device) to reformat the NTFS-formatted external drive to ext3. I set the label to 'SeagateGoflex320'. (I entered 'SeagateGoflex320GB' but, apparently, only the first 16 characters were used.) Then I was able to mount the external drive by unplugging the USB connection and reconnecting. I felt much relieved at this point. I could use the drive for something other than a paper-weight. I then used shell-scripting to setup a 'fileCleanup' icon and a 'bkup_2Seagate320GB' icon on my desktop, just below the button on the right of the upper Ubuntu/Gnome2-Nautilus 'top-panel'. Now I was prepared to do the following. Before I click on the 'top-panel' button for the LockScreen/Suspend/Hibernate/Restart/Shutdown options, to shutdown the netbook, if I have edited some files in my home directory, I click on the 'fileCleanup' icon to cleanup files before the backup. Then I connect the backup drive and I use the 'bkup' icon to rsync-backup my files. Works like a charm --- after that initial mount-problem scare. Thank you Google developers, Ubuntu developers and users, fdisk developers, and Gparted developers. In addition to being thankful for my family, I felt I had to add these thank-you's in the 2010 Thanksgiving season in which this near-fiasco occurred. |
Bottom of this To return to a previously visited web page location, click on the Back button of your web browser a sufficient number of times. OR, use the History-list option of your web browser. OR ...
< Go to start of Code samples, above. >
Page history:
Page was created 2021 Jun 20.
|