I don't think there's anything "tricky" in WCF or the .NET framework to do this, really. What are you passing back to the client? An instance of a class?
What you could do is run the query or however you fetch the response, and then serialize that into a memory stream and see how big it gets. This won't be a totally accurate size - the SOAP messages has some overhead to it, like SOAP envelope and headers and stuff - but it can give you a ballpark figure of whether you're about to return a few hundred bytes, or a couple megabytes.
Trouble is: this might take a while on the server just to assemble / query, and then to actually "measure", too. Plus you'd almost have to have two calls - one to "MeasureResult" which returns an Int or Long or something, and then a second call "GetResult" to actually get the results. So you'll incur that effort to assemble the message twice....
I don't really have a good answer for you, but maybe you just need to figure out some other way to allow the client to abort a call if it takes too long. Or find a way to more quickly figure out an indicator as to how large the response will be (without getting all the details of the response itself).