Why is the size of files capped at 4 GB when outputting to a file using print? I would expect that with streaming output it should be possible to generate files of arbitrary size.
Update: ijw and Chas. Owens were correct. I thought the F: drive was NTFS formatted, but in fact it used the FAT32 filesystem. I tried it on another drive and I could generate a 20 GB text file. There are no limits in this case. Apologies to all.
Details: while researching for answering a question here on Stack Overflow I needed to measure the performance of reading a very large text file using Perl. In order to test the reading I needed a large text file and I wrote a small Perl script to generate the text file and ran into an unexpected problem. The output file grows until it reach 4 GB. According to Windows Explorer the size in one run of the script was 4294967269 bytes (and 4294967296 bytes on disk). The script continues, but the file no longer grows.
Essential it is just a number of:
print NUMBERS_OUTFILE $line;
where $line is a long string with a "\n" at the end. The length of the line can be configured and is not critical for this problem; e.g. 250 characters or 34000 characters. NUMBERS_OUTFILE is a file handle created with:
open ( NUMBERS_OUTFILE,">F:\temp2\out1.txt")
Drive F: is NTFS formatted and is on a separate physical hard disk from the disk with the operating system.
What is the reason and is there a work-around?
Full Perl script and BAT driver script (HTML formatted with the pre tag). If the two environment variables MBSIZE and OUTFILE are setup then the Perl script should be able to run unchanged on other platforms than Windows.
Platform: Perl 5.10.0 from ActiveState; 32 bit; build 1004. Windows XP x64 SP2, 8 GB RAM, 500 GB Green Caviar hard disks.
perl -V says:
Summary of my perl5 (revision 5 version 10 subversion 0) configuration:
Platform:
osname=MSWin32, osvers=5.00, archname=MSWin32-x86-multi-thread
uname=''
config_args='undef'
hint=recommended, useposix=true, d_sigaction=undef
useithreads=define, usemultiplicity=define
useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
use64bitint=undef, use64bitall=undef, uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='cl', ccflags ='-nologo -GF -W3 -MD -Zi -DNDEBUG -O1 -DWIN32 -D_CONSOLE -DNO_ST
RICT -DHAVE_DES_FCRYPT -DUSE_SITECUSTOMIZE -DPRIVLIB_LAST_IN_INC -DPERL_IMPLICIT_CONTE
XT -DPERL_IMPLICIT_SYS -DUSE_PERLIO -DPERL_MSVCRT_READFIX',
optimize='-MD -Zi -DNDEBUG -O1',
cppflags='-DWIN32'
ccversion='12.00.8804', gccversion='', gccosandvers=''
intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
d_longlong=undef, longlongsize=8, d_longdbl=define, longdblsize=10
ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='__int64', lseeksize=8
alignbytes=8, prototype=define
Linker and Libraries:
ld='link', ldflags ='-nologo -nodefaultlib -debug -opt:ref,icf -libpath:"D:\Perl\
lib\CORE" -machine:x86'
libpth=\lib
libs= oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib a
dvapi32.lib shell32.lib ole32.lib oleaut32.lib netapi32.lib uuid.lib ws2_32.lib mpr.l
ib winmm.lib version.lib odbc32.lib odbccp32.lib msvcrt.lib
perllibs= oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.l
ib advapi32.lib shell32.lib ole32.lib oleaut32.lib netapi32.lib uuid.lib ws2_32.lib m
pr.lib winmm.lib version.lib odbc32.lib odbccp32.lib msvcrt.lib
libc=msvcrt.lib, so=dll, useshrplib=true, libperl=perl510.lib
gnulibc_version=''
Dynamic Linking:
dlsrc=dl_win32.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' '
cccdlflags=' ', lddlflags='-dll -nologo -nodefaultlib -debug -opt:ref,icf -libpat
h:"D:\Perl\lib\CORE" -machine:x86'
Characteristics of this binary (from libperl):
Compile-time options: MULTIPLICITY PERL_DONT_CREATE_GVSV
PERL_IMPLICIT_CONTEXT PERL_IMPLICIT_SYS
PERL_MALLOC_WRAP PL_OP_SLAB_ALLOC USE_ITHREADS
USE_LARGE_FILES USE_PERLIO USE_SITECUSTOMIZE
Locally applied patches:
ActivePerl Build 1004 [287188]
33741 avoids segfaults invoking S_raise_signal() (on Linux)
33763 Win32 process ids can have more than 16 bits
32809 Load 'loadable object' with non-default file extension
32728 64-bit fix for Time::Local
Built under MSWin32
Compiled at Sep 3 2008 13:16:37
@INC:
D:/Perl/site/lib
D:/Perl/lib
.