What is a good method for making a huge test file?

Question

I'm trying to test some streaming file upload/download code that I've just written, but I need a huge (bigger than my memory, as in > 4GB) file to test on. What's a good way to quickly generate a file of a specific size (but no particular data inside)?

I know that I could just look around my hard drive for an appropriately-sized file, but I'm curious if there are any other good ways for generating test files, especially if I want a specific size. Downloading files is no good either, since my connection is slow.

Answer 1

On Unix this is easy:

head -c <numbytes> /dev/random > bigfile.dat

So for 4 gigabytes this would be approximately:

head -c 4294967296 /dev/random > bigfile.dat

Answer 2

On a *nix platform, you could do something like this:

dd if=/dev/zero of=test5000M.bin bs=5000000000 count=1

Which would generate a 5000mb file full of nothing.

On windows, something like this should work:

fsutil file createnew test5000M.bin 5000000000

Answer 3

I used to just do a seek and a write past the end of the file, to allocate huge chunks.

However, depending on the size you use, this can fail. (Near the 1.2 TB limit, in my experience on WinXP)

I used this method successfully for a hard-drive burn-in application for quite a few different system configurations.

Answer 4

On *nix systems, you could create a sparse file that hardly uses any real disk space.

dd if=/dev/zero bs=1 seek=4294967295 count=1 of=4G
ls -l 4G
-rw-r--r-- 1 me wheel 4294967296 2009-04-01 13:28 4G

This creates a file with a hole of size 4GB - 1 in the beginning. The hole does not consume filesystem blocks. You can even create sparse files that are too big to fit in your filesystem. Reading from the hole returns zeroes.

Answer 5

Here's a function to do it in Windows PowerShell:

function Create-TestFile([string]$location, [long]$sizeInBytes, [int]$numberOfFiles = 1 ,[char]$fillerChar="X")
{
$fc = new-object string ($fillerChar, $sizeInBytes ) ;
1..$numberOfFiles | %{ [io.file]::WriteAllText("$location\TestFile-$_.txt", $fc) }
}

And here's how you'd use it:

Create-TestFile "C:\temp" 2100 #creates a file 2100 bytes in size.

Answer 6

On Windows you can also use the fsutil command:

Usage : fsutil file createnew <filename> <length>
   Eg : fsutil file createnew C:\testfile.txt 1000

This will create a file called testfile.txt with size of 1,000 bytes. The file just contains NULs though, if that matters.

Answer 7

I used to use an eatbits.exe utility back in the day. Where you could create these files. They are very valuable for installation testing.

I couldn't find eatbits anywhere online, it might have been an internal utility at the company I was working for.