Are there limitations on the amount of data you can redirect to a file under Windows (XP/2003)?

Question

I'm driving a web services load test using a system of WSF, VBS and JavaScript scripts. The load driver is running on my workstation, sending requests to a remote application server. I'm using WScript.StdOut.Write in the driver script to write results, and just redirecting the output to a text file when I run the test:

cscript //nologo driver.wsf > test_results.txt

When I use my "smoke test" input file of only 100 transactions, the test runs fairly quickly, and without issues. When I use my "load test" input file of 200,000 transactions, the performance of the driver degrades over time, to the point of the workstation on which it's running becoming unresponsive and is using 85% of the page file. The size of my test_results.txt file was just over 43GB.

I suspect that Windows is caching the StdOut output to memory rather than writing it to the file; does anyone have an alternative explanation or other ideas? Would it be better to manage the output as file system objects from my script rather than use StdOut?

UPDATE: My driver basically does this (psuedo code):

Open input file 
Read a record 
While not EOF 
    Encode record 
    Create SOAP message containing record 
    Make web service request 
    Write time to get response to StdOut 
    Read another record 
End While 
Close input file

Answer 1

2GB is file size limit on windows.

Answer 2

I doubt very seriously that the problem is with the redirected output. I often create redirected StdOut files that are hundreds of megabytes in size.

More likely, something about encoding the record, creating the SOAP message, or making the Web service request is leaking memory.

You might consider commenting out the line that writes the response time to StdOut, and re-run the program. If the program still begins slowing down, then you've eliminated the output as the cause.