If you already know that the file being requested might be quite large, then you might want to create your own specific endpoint for this request, which supports streaming.
So you would have a regular endpoint (e.g. http://yourserver:8080/YourService) which you use for "normal" method calls, and a second endpoint (http://yourserver:8085/YourService) which would support streaming to send back the file with a reasonable amount of memory overhead.
Configuring this should be fairly simple - both on the server and the client, you need to specify a binding configuration to support streaming:
<system.serviceModel>
<bindings>
<basicHttpBinding>
<binding name="streamed"
transferMode="StreamedResponse" />
</basicHttpBinding>
</bindings>
<services>
<service name="YourService">
<endpoint name="normal"
address="http://yourserver:8080/YourService"
binding="basicHttpBinding"
contract="IYourServiceContract" />
<endpoint name="filetransfer"
address="http://yourserver:8085/YourService"
binding="basicHttpBinding"
bindingConfiguration="streamed"
contract="IYourServiceContract" />
</service>
</services>
</system.serviceModel>
On the client, of course, you'd have to have the two endpoints inside a <client>
tag, but otherwise, everything should be the same.
The "transferMode" is "buffered" by default, e.g. the whole message is buffered and sent in one block.
Your other options are "Streamed" (streaming both ways), "StreamedRequest" (if you have really large requests) or "StreamedResponse" (if only the response, the file being transferred, is really large).
In this case, you'd have a single method on your service that would return a stream (i.e. the file). From your client, when you call this service method, you get back a stream that you can then read in chunks, just like a MemoryStream or a FileStream.
Marc