Fast Disk Cloning

Question

Is there a way to have Linux read ahead when cloning a disk? I use the program named "dd" to clone disks. The last time I did this it seemed as though the OS was reading then writing but never at the same time. Ideally, the destination disk would be constantly writing without waiting that's of course if the source disk can keep up.

UPDATE: I normally choose a large block size when cloning (ex. 16M or 32MB).

Answer 1

Maybe you can use two processes:

dd if=indevfile | dd of=outdevfile

I'll assume you can set the other dd options as it suits you. This has some overhead but should allow asynchrony between reading one disk and writing the other.

Answer 2

Are you sure it isn't doing that at the same time? I would expect the disk caches to make sure it that happens. If not, non-blocking or even asynchronous reads/writes may help,

Answer 3

You might try increasing the block size using the bs argument; by default, I believe dd uses a block size equal to the disk's preferred block size, which will mean many more reads and writes to copy an entire disk. Linux's dd supports human-readable suffixes:

dd if=/dev/sda of=/dev/sdb bs=1M

Answer 4

Possibly silly question, but are these disks on the same controller/channel etc? some controllers aren't very good at concurrent operations.

Answer 5

About your update: How big are the caches of your HDs? (specially the writing one). It may be that that is too much and you may need to reduce it to prevent unnecessary blocking.

Answer 6

Commodore Jaeger is right about:

dd if=/dev/sda of=/dev/sdb bs=1M

Also, adjusting "readahead" on the drives usually improves performance. The default may be something like 256, and optimal 1024. Each setup is different, so you would have to run benchmarks to find the best value.

# blockdev --getra /dev/sda
256
# blockdev --setra 1024 /dev/sda
# blockdev --getra /dev/sda
1024
# blockdev --help
Usage:
  blockdev -V
  blockdev --report [devices]
  blockdev [-v|-q] commands devices
Available commands:
    --getsz (get size in 512-byte sectors)
    --setro (set read-only)
    --setrw (set read-write)
    --getro (get read-only)
    --getss (get sectorsize)
    --getbsz    (get blocksize)
    --setbsz BLOCKSIZE  (set blocksize)
    --getsize   (get 32-bit sector count)
    --getsize64 (get size in bytes)
    --setra READAHEAD   (set readahead)
    --getra (get readahead)
    --flushbufs (flush buffers)
    --rereadpt  (reread partition table)
    --rmpart PARTNO (disable partition)
    --rmparts   (disable all partitions)
#

Answer 7

You use really big Block size. I normally use 32256. I got this number from a blog writen in 2005. I want to know what shoud I use now. Do you think using 16M, 32M will significantly decrease disc cloning time?