'Big Copy' of files
|
|
A few more notes and images may be added,
if/when I revisit this page in the future.
|
INTRODUCTION : In doing a copy of a 'big mass' of files (several Gigabytes) to a USB 'stick', I ended up with a ruined USB stick. Whenever I tried to use the USB stick by sticking it into a USB slot on my computer, the mount of the USB stick 'hung' --- as indicated in the following several images.
|
Note the 'mounting' message at the upper part of
this 'Palimpsest' Disk Utility GUI. The progress
indicator just kept cycling interminably.
|
Let me back up here and describe the environment that I was working in. I was using a Linux operating system --- Ubuntu 9.10 (the 2009 October release, 'Karmic Koala') --- with the Gnome 2 desktop environment. In the Gnome 2 desktop, there is a 'panel' at the top of the screen with 'Applications Places System' drop-down menu options. In particular, the 'Places' menu shows a 'Computer' line-item followed by any 'auto-mounted' drives on your machine, such as USB sticks or external USB hard drives. For example, see the image at the top of this page. NOTE: Gnome2 being replaced by MATE. Around 2010, the Gnome2 environment was no longer being developed by the Gnome group. The Gnome 3 environment threw out a lot of the Gnome 2 code and started on a new type of desktop environment, oriented toward tablets-and-fingers rather than desktop-computers-and-mice. But a group (started in Argentina) 'forked' the Gnome2 desktop environment and Nautilus file manager into the MATE (MAH-tay - a coffee-like drink) desktop environment and the Caja (CAH-hah - Spanish for 'box') file manager. In the future (2015 and beyond), since the Gnome2-Nautilus environment is no longer available in Linux distros, I may migrate to a Linux operating system using MATE-Caja --- such as Linux Mint MATE or Ubuntu MATE. Preparing (formatting) USB sticks When I use USB 'sticks' for copies of my files (PDF files, image files, audio files, video files, whatever), I often reformat the sticks --- from the 'FAT32' file system with which they are usually formatted --- to a Linux file format such as 'ext3'. I have described my typical USB-drive reformatting process on this Linux Disk Formatting web page. A GUI-based USB-stick formatting process is indicated in the following several images. The 'Places' menu in the Gnome2 'top panel' offers a 'Computer' line-item, as seen in the following image.
|
Here I right-clicked on the Lexar USB line-item, and I chose
'Format' from the popout menu of drive options.
The 'Format' dialog window appeared.
I chose a file-system format and provided
a meaningful 'volume ID' for the drive.
|
The 'Big Copy' Problem (and solution) When I am going to copy a 'large mass' of files (say PDF files) to a USB stick, I typically plug the 'to' stick into a USB slot on my computer and bring up the Nautilus file manager to show the contents of the (formatted) USB stick. The way I do this: When I plug the USB stick into a USB slot on the computer, within about 20 seconds an icon representing the stick shows up on the desktop. I right-click on the icon and choose 'Open'. The Nautilus file manager starts up showing the contents (directories and other files) at the top-level of the USB stick. The first time I tried copying about 3 Gigabytes of PDF files onto a USB stick, I ended up with an unusable USB stick. I ended up with a USB stick that would not mount --- as indicated in the two images at the start of the 'Introduction' above. To do a copy of a directory of files, I typically right-click-Copy on a directory in another Nautilus file manager window (opened on a hard-drive or USB-stick directory) and Paste the selected directory into the Nautilus file manager window opened on a directory level of the 'to' USB stick. Here is where I get to the 'corrupted' USB stick after copying about 3 Gig of PDF files to a 16 Gig USB stick. After I did the Paste of the 3 Gig directory into a directory of the 'to' USB stick, a copy-progress window opened up, by which I could monitor the copy process. See the following image of a 'File Operations' window. |
|
When this progress bar filled to the right and the window automatically closed, I thought the copy process was complete. Boy, was I wrong. I should have known something was incomplete because when I did a 'Shutdown' of my computer soon after this copy, the shutdown did not complete within a few seconds as it normally does. Instead, a black screen appeared with a blinking cursor in the upper-lefthand corner. I thought for sure the copy had completed, so I shutdown the computer with the power switch on the computer. I found out later that I had just corrupted the 'to' USB stick. After this interrupted shutdown is when I encountered the 'mount hangs' as seen in the two 'mount' images near the top of the 'Introduction'. In trying to find if I could recover the use of the USB stick, I did Google searches on keywords such as 'usb stick mount problem "fdisk hangs"'. I knew from a situation I documented on my Linux Disk Formatting web page that you can usually see the disk drives mounted on your computer by using the 'sudo fdisk -l' command. But at the point where the 'fdisk' command would ordinarily show the info on the USB stick, the 'fdisk' command would hang when the corrupted USB stick was in a USB slot on the computer. In doing my Google searches, I ran across a 5-page forum thread titled 'USB file copy very slow' that had some interesting and useful comments --- that suggested what caused my USB stick to be 'fried'. I extract below the comments (posts) that had the most meaning to me.
Following are some interesting notes on 'USB file copy very slow' from
Arch Linux forum pages at I have high-lighted in bold font the phrases that were particularly meaningful and useful for me. 1) In post #74 on page 3, 'Aidenn' points out: let's start with the fact, that people are experiencing three different issues here. first one is the transfer "starts fast, then slows down". it's a normal async behaviour, at first the file is copied to cache, and after the cache fills up, the real transfer speed shows up. if the file is small enough, it may not get past the cache at all, so it appears to be copied instantly. after "finishing" the transfer, the system still has to sync the cache with the stick, that's why you can't unmount the stick. just wait or use -o sync, so you won't see the fake speed at the beginning. second one is the general slowness. well, ~400kB/s is really bad and looks like a bug, but ~2MB/s is pretty normal, depending on the stick. Standard write speeds for modern sticks are somewhere between 2 and 6 MB/s (even if the manufacturer says, for example, 11MB/s). it gets even worse with big files or a lot of small ones. it's OS-independent, and you can't do anything about it. third one is slow-only-in-Arch. I can think of two things - first, other OSes have better caching schemes, but it doesn't mean that it's really faster. and you can lose your data if you take the stick out after the transfer seemingly ended, but the cache still hasn't synced. second is that there really is something wrong with the Arch kernel, but everything works fine for me. I'm getting ~4MB/s with my Goodram Twister 4GB, same as in Windows. ... 2) In post #78 on page 4, 'Aidenn' points out: "unreadable" files [in 'demsg' kernel log output] mean that the disk was unplugged prematurely in most cases. maybe some other form of file corruption took place, FAT filesystem is notoriously known for its corruptability. for that tool you wanted, assuming your stick is FAT16/32 formatted, use dosfsck (or chkdsk if you have a windows machine). NOTE: I was seeing "unreadable" messages in 'demsg' output when I inserted the corrupted USB stick in a USB slot of my computer. 3) In post #80 on page 4, 'Aidenn' points out: ... sync [in a mount command for the USB stick] isn't that important, it doesn't change the real speed. just remember to unmount the stick before you plug it out, or use the 'sync' command (when it goes back to the command prompt, it means you can unplug). I don't know if XFCE does it, but Gnome has a little popup when you click unmount, which tells you to wait for the flush and disappears when it's safe to unplug. with '-o sync' it's a bit more user-with-ADHD-resistant, but ultimately nothing really important. NOTE: Problem is ... I did not know that it might take several minutes, after the 'formatting progress window' closed, for the cache to be 'flushed' to the USB stick --- THEN allowing the USB stick to be unmounted. Thanks to 'Aidenn', I now know that I can use the 'sync' command in a terminal window to determine when the copy to the USB stick is truly finished. This is better than watching for the light on the 'to' USB stick to stop blinking. 4) In post #102 on page 5, 'alleyoopster' points out results from 2 speed tests: TEST#1: Copying 621MB iso to USB stick - 189 seconds (3.2 MB/s) I.e. about 3.2 min for 0.62 Gig --- or about 52 minutes per 10 Gigabytes. TEST#2: |
# dd if=/dev/zero of=/media/Quatre/tmp/output.img bs=8k count=256k
262144+0 records in
262144+0 records out
2147483648 bytes (2.1 GB) copied, 415.704 s, 5.2 MB/s
|
I.e. about 6.5 min for 2.1 Gig --- or about 31 minutes per 10 Gigabytes. He began by saying: "I haven't got hectic slow speeds, but I am only getting 3MB/s write on a contiguous file, which I would say is slow for USB 2.0 Is it unreasonable to expect 12MB/s?" Actually, as 'Aidenn' pointed out above, about 3 MB/sec (about 55 minutes for 10 Gigabytes) is normal for USB 2.0. 5) In post #118 on page 5, 'thisoldman' points out a kernel developer's comments on 'flash drives': An article on lwn.net from February 2011, by Arnd Bergmann, a kernel developer, says that a cure for slow transfers won't be seen soon: "Optimizing Linux with cheap flash drives" - http://lwn.net/Articles/428584/. This article, after a very technical discussion of the architecture and file I/O processing in flash drives, points out that "There is a huge potential for optimizing Linux to better deal with ... flash media in various places in the kernel and elsewhere." My Observations on 'Big Copy'
individual files) from one 16GB pen drive to another Here are some images from copying a couple of large directories to a 16 Gig USB stick --- a directory called 'PDFs' (about 3.1 Gig) and a directory called 'misc' (about 6.5 Gig). |
Here is the start of copying the 'PDFs' directory.
Here I thought the 'PDFs' copy was finished,
so I started a copy of the 'misc' directory.
Note that the Nautilus 'File Operations'
window supports multiple progress bars for
multiple copy tasks that are proceeding simultaneously.
Here the copy of the 'misc' directory is ending
at a rate of about 1.8 Megabytes/second.
|
I make several observations below, based on my experiences --- and from what 'Aidenn' said above. 1) I ruined a USB drive by interrupting the multiple-minute process of 'flushing' cache that was being written to the drive after the 'File Operations' window had closed. The copying APPEARED to be done. (Maybe it would have been clearer that the copy was not yet complete if I had used the 'cp' copy command in a terminal window, rather than using the Nautilus file manager GUI's. The progress GUI window should not close until the cache is flushed.) I chose to shutdown the machine but it would not complete the shutdown. A blank screen with only a blinking cursor in the upper left greeted me --- instead of the usual power-down of the computer. I realized later that the shutdown procedure detected that there was this process that needed to complete --- but it did not give me any user-friendly messages to let me know that I needed to wait (perhaps several minutes --- maybe even 10 to 20 minutes) for the process to complete. Lesson: If you get the blinking cursor on shutdown, walk away and leave it alone. Check back in about half an hour to see if the shutdown completed. This USB stick is unusable. Whenever I put it in a USB slot (after logging in), it shows up in a list of drives (in the 'Places' drop down menu) --- but holding a cursor over that drive ID/icon shows a popup that says 'Mount <Volume-ID-goes-here>'. Clicking on the ID/icon causes an 'Error' window to popup with the message: |
Unable to mount <Volume-ID-goes-here>
Dbus error org.gtk.Private.RemoteVolumeMonitor.Failed:
An operation is already pending.
|
Here is the actual 'Error' window:
'sudo fdisk -l' hangs and does not show the USB stick as mounted. I have not found a way to mount the drive and see if I can see the directories and files on the drive --- or, at least, do a 'reformat' of the corrupted USB stick. Also, when I try the Palimpsest 'Disk Utility' (under System > Administration), it shows the USB drive is in a mount operation --- and a Cancel button is grayed out --- so I cannot try to stop the mount process. |
|
The light on the USB drive is blinking, but the mount is not proceeding successfully. Apparently the light will blink forever if I let it --- or until mount I/O error messages in the 'demsg' kernel log consume the space in a file system and the operating system freezes up. 2) Although the initial display in the 'File Operations' window showed copy speed of about 6 to 7 MB/sec, by the middle to the end of the copying the speed typically settled down to about 2 to 4 MB/sec. 3) Takes about 3 to 5 minutes to copy each Gig of files at a 'nominal rate' of about 2 to 4 Megabytes per second --- where 'nominal rate' is the rate shown in the 'File Operations' window. (This 3 to 5 min/GB rate makes sense --- especially if you add on the time that it takes to 'flush' the cache. See items 4 and 5 below.) 4) After about an hour copy, it took another 20 to 30 minutes (of the 'to' drive blinking) for the cached files to be copied to the 'to' drive and for the drive to stop blinking. It was then ready for a 'Safely Remove Drive' operation. 5) It is not always clear when cache is still emptying to the 'to' drive (especially if you cannot see a light blinking on the 'to' USB drive). As added confirmation that 'cache-flushing' is done, you can enter the 'sync' command in a terminal window. The command prompt does not re-appear until the caches of the various drives have been 'flushed'. 6) Someday I may add some speed results here for a USB 3.0 drive --- when I can assure that the entire chain of hardware and software (USB stick, USB slot on the computer, kernel software, and whatever else) will handle 3.0 --- and not revert to 2.0 protocol. BOTTOM LINE: USB 'sticks' may be OK for copying small amounts of files (less than 1 Gig) at a time --- say for 'sneaker net' movement of files from one computer to another. But for backing up 10's or 100's of Gigabytes of files, it is probably better to use 'external' USB disk drives --- or, better yet, for speed, second hard drives permanently mounted in the computer. --- Furthermore, use the 'sync' command BEFORE powering down --- after copying lots of files (or a huge file) to a USB stick --- and maybe after 'big-copying' to any USB storage drive --- or any storage drive. |
|
Bottom of this
To return to a previously visited web page location, click on the
Back button of your web browser a sufficient number of times.
OR, use the History-list option of your web browser.
< Go to Top of Page, above. >
|