Logging raw HTTP request/response in ASP.NET MVC & IIS7

Question

I'm writing a web service (using ASP.NET MVC) and for support purposes we'd like to be able to log the requests and response in as close as possible to the raw, on-the-wire format (i.e including HTTP method, path, all headers, and the body) into a database.

What I'm not sure of is how to get hold of this data in the least 'mangled' way. I can re-constitute what I believe the request looks like by inspecting all the properties of the HttpRequest object and building a string from them (and similarly for the response) but I'd really like to get hold of the actual request/response data that's sent on the wire.

I'm happy to use any interception mechanism such as filters, modules, etc. and the solution can be specific to IIS7. However, I'd prefer to keep it in managed code only.

Any recommendations?

Edit: I note that HttpRequest has a SaveAs method which can save the request to disk but this reconstructs the request from the internal state using a load of internal helper methods that cannot be accessed publicly (quite why this doesn't allow saving to a user-provided stream I don't know). So it's starting to look like I'll have to do my best to reconstruct the request/response text from the objects... groan.

Edit 2: Please note that I said the whole request including method, path, headers etc. The current responses only look at the body streams which does not include this information.

Edit 3: Does nobody read questions around here? Five answers so far and yet not one even hints at a way to get the whole raw on-the-wire request. Yes, I know I can capture the output streams and the headers and the URL and all that stuff from the request object. I already said that in the question, see:

I can re-constitute what I believe the request looks like by inspecting all the properties of the HttpRequest object and building a string from them (and similarly for the response) but I'd really like to get hold of the actual request/response data that's sent on the wire.

If you know the complete raw data (including headers, url, http method, etc.) simply cannot be retrieved then that would be useful to know. Similarly if you know how to get it all in the raw format (yes, I still mean including headers, url, http method, etc.) without having to reconstruct it, which is what I asked, then that would be very useful. But telling me that I can reconstruct it from the HttpRequest/HttpResponse objects is not useful. I know that. I already said it.

---

Please note: Before anybody starts saying this is a bad idea, or will limit scalability, etc., we'll also be implementing throttling, sequential delivery, and anti-replay mechanisms in a distributed environment, so database logging is required anyway. I'm not looking for a discussion of whether this is a good idea, I'm looking for how it can be done.

Answer 1

HttpRequest and HttpResponse pre MVC used to have a GetInputStream() and GetOutputStream() that could be used for that purpose. Haven't look into those part in MVC so Im not sure they are availavle but might be an idea :)

Answer 2

use a IHttpModule:

    namespace Intercepts
{
    class Interceptor : IHttpModule
    {
     private readonly InterceptorEngine engine = new InterceptorEngine();

     #region IHttpModule Members

     void IHttpModule.Dispose()
     {
     }

     void IHttpModule.Init(HttpApplication application)
     {
      application.EndRequest += new EventHandler(engine.Application_EndRequest);
     }
     #endregion
    }
}

    class InterceptorEngine
    {  
     internal void Application_EndRequest(object sender, EventArgs e)
     {
      HttpApplication application = (HttpApplication)sender;

      HttpResponse response = application.Context.Response;
      ProcessResponse(response.OutputStream);
     }

     private void ProcessResponse(Stream stream)
     {
      Log("Hello");
      StreamReader sr = new StreamReader(stream);
      string content = sr.ReadToEnd();
      Log(content);
     }

     private void Log(string line)
     {
      Debugger.Log(0, null, String.Format("{0}\n", line));
     }
    }

Answer 3

Agree with FigmentEngine, IHttpModule appears to be the way to go

look into httpworkerrequest readentitybody and GetPreloadedEntityBody

to get the httpworkerrequest you need to do this:

(HttpWorkerRequest)inApp.Context.GetType().GetProperty("WorkerRequest", bindingFlags).GetValue(inApp.Context, null);

where inApp is the httpapplication object...

Answer 4

I know it's not managed code, but I'm going to suggest an ISAPI filter. It's been a couple of years since I've had the "pleasure" of maintaining my own ISAPI but from what I recall you can get access to all this stuff, both before and after ASP.Net has done it's thing.

http://msdn.microsoft.com/en-us/library/ms524610.aspx

If a HTTPModule isn't good enough for what you need, then I just don't think there is any managed way of doing this in the required amount of detail. It's gonna be a pain to do though.

Answer 5

I agree with the others, use an IHttpModule. Take a look at the answer to this question, which does almost the same thing that you are asking. It logs the request and response, but without headers.

How to trace ScriptService WebService requests?

Answer 6

To FigmentEngine: response.OutputStream has CanRead property as False (framework 3.5), so it can not be read.

Answer 7

Well, I'm working on a project and did, maybe not too deep, a log using the request params:

Take a look:

public class LogAttribute : ActionFilterAttribute
{
    private void Log(string stageName, RouteData routeData, HttpContextBase httpContext)
    {
        //Use the request and route data objects to grab your data
        string userIP = httpContext.Request.UserHostAddress;
        string userName = httpContext.User.Identity.Name;
        string reqType = httpContext.Request.RequestType;
        string reqData = GetRequestData(httpContext);
        string controller = routeData["controller"];
        string action = routeData["action"];

        //TODO:Save data somewhere
    }

    //Aux method to grab request data
    private string GetRequestData(HttpContextBase context)
    {
        StringBuilder sb = new StringBuilder();

        for (int i = 0; i < context.Request.QueryString.Count; i++)
        {
            sb.AppendFormat("Key={0}, Value={1}<br/>", context.Request.QueryString.Keys[i], context.Request.QueryString[i]);
        }

        for (int i = 0; i < context.Request.Form.Count; i++)
        {
            sb.AppendFormat("Key={0}, Value={1}<br/>", context.Request.Form.Keys[i], context.Request.Form[i]);
        }

        return sb.ToString();
    }

You can decorate your controllers class for log it entirely:

[Log]
public class TermoController : Controller {...}

or log just some individual action methods

[Log]
public ActionResult LoggedAction(){...}

Answer 8

Definitely use an IHttpModule and implement the BeginRequest and EndRequest events.

All of the "raw" data is present between HttpRequest and HttpResponse, it just isn't in a single raw format. Here are the parts needed to build Fiddler-style dumps (about as close to raw HTTP as it gets):

request.HttpMethod + " " + request.RawUrl + " " + request.ServerVariables["SERVER_PROTOCOL"]
request.Headers // loop through these "key: value"
request.InputStream // make sure to reset the Position after reading or later reads may fail

For the response:

"HTTP/1.1 " + response.Status
response.Headers // loop through these "key: value"

Note that you cannot read the response stream so you have to add a filter to the Output stream and capture a copy.

In your BeginRequest, you will need to add a response filter:

HttpRequest request = HttpContext.Current.Request;
OutputFilterStream filter = new OutputFilterStream(request.Filter);
request.Filter = filter;

Store filter where you can get to it in the EndRequest handler. I suggest in HttpContext.Items. There can then get the full response data in filter.ToString().

Then implement OutputFilterStream using the Decorator pattern as a wrapper around a stream:

/// <summary>
/// A stream which keeps an in-memory copy as it passes the bytes through
/// </summary>
public class OutputFilterStream : Stream
{
    private readonly Stream InnerStream;
    private readonly MemoryStream CopyStream;

    public OutputFilterStream(Stream inner)
    {
     this.InnerStream = inner;
     this.CopyStream = new MemoryStream();
    }

    public string ReadStream()
    {
     lock (this.InnerStream)
     {
      if (this.CopyStream.Length <= 0L ||
       !this.CopyStream.CanRead ||
       !this.CopyStream.CanSeek)
      {
       return String.Empty;
      }

      long pos = this.CopyStream.Position;
      this.CopyStream.Position = 0L;
      try
      {
       return new StreamReader(this.CopyStream).ReadToEnd();
      }
      finally
      {
       try
       {
        this.CopyStream.Position = pos;
       }
       catch { }
      }
     }
    }


    public override bool CanRead
    {
     get { return this.InnerStream.CanRead; }
    }

    public override bool CanSeek
    {
     get { return this.InnerStream.CanSeek; }
    }

    public override bool CanWrite
    {
     get { return this.InnerStream.CanWrite; }
    }

    public override void Flush()
    {
     this.InnerStream.Flush();
    }

    public override long Length
    {
     get { return this.InnerStream.Length; }
    }

    public override long Position
    {
     get { return this.InnerStream.Position; }
     set { this.CopyStream.Position = this.InnerStream.Position = value; }
    }

    public override int Read(byte[] buffer, int offset, int count)
    {
     return this.InnerStream.Read(buffer, offset, count);
    }

    public override long Seek(long offset, SeekOrigin origin)
    {
     this.CopyStream.Seek(offset, origin);
     return this.InnerStream.Seek(offset, origin);
    }

    public override void SetLength(long value)
    {
     this.CopyStream.SetLength(value);
     this.InnerStream.SetLength(value);
    }

    public override void Write(byte[] buffer, int offset, int count)
    {
     this.CopyStream.Write(buffer, offset, count);
     this.InnerStream.Write(buffer, offset, count);
    }
}

Answer 9

OK, so it looks like the answer is "no you can't get the raw data, you have to reconstruct the request/response from the properties of the parsed objects". Oh well, I've done the reconstruction thing.

Answer 10

I have a question on the code posted so far. I am trying to get a copy of what is normally output to the web browser to an html file that I will create from the same http response output so I can email the page (a complex report).

I need to access the output after the content is rendered. I am doing this in my MVC app so my authenticated users can mail certain reports to third parties that don't have access to the application and cannot access the original report.

When I tried using the following in my an http module per FigmentEngine above:

 internal void Application_EndRequest(object sender, EventArgs e)
    { …
       HttpResponse response = application.Context.Response;
        ProcessResponse(response.OutputStream);
      …
    }
private void ProcessResponse(Stream stream) {
            StreamReader sr = new StreamReader(stream);
      …
}

I get the error ”System.ArgumentException: Stream was not readable.”

Since this stream isn’t readable I decided to use a copy. I am using the OutputFilterStream and filter suggested by McKAMEY above.

namespace HttpIntercepts
{
    class InterceptorModule : IHttpModule
    {
        private readonly InterceptorEngine engine = new InterceptorEngine();

       #region IHttpModule Members
        void IHttpModule.Dispose()
        {
        }

        void IHttpModule.Init(HttpApplication application)
        {
            application.BeginRequest += new EventHandler(engine.Application_BeginRequest);
            application.EndRequest += new EventHandler(engine.Application_EndRequest);
        }
        #endregion
    }
}

class InterceptorEngine
{
    internal void Application_BeginRequest(object sendder , EventArgs e)
    {        
        HttpRequest request = HttpContext.Current.Request;
        OutputFilterStream filter = new OutputFilterStream(request.Filter);
        request.Filter = filter;
        HttpContext.Current.Items.Add("requestfilter", filter);
    }

    internal void Application_EndRequest(object sendder , EventArgs e)
    {        
        HttpApplication application = (HttpApplication)sender;
        HttpResponse response = application.Context.Response;
        ProcessResponse(response.OutputStream); // process the http response
    }

    private void ProcessResponse(Stream stream)
    {
        HttpRequest request = HttpContext.Current.Request;
        OutputFilterStream filter = (OutputFilterStream)HttpContext.Current.Items["filter"];
       //... need code here to get a string that would contain the response
       //...change css and js file paths
       //...save to html file
       //...email
    }
}

What do I need to do in ProcessResponse to get the response output into a string so I can write that into an email and send it?

What I am looking for is a string that will look like what you see when you view source of the page.

TIA,

Steve

Answer 11

You can use the ALL_RAW server variable to get the original HTTP headers sent with the request, then you can get the InputStream as usual:

string originalHeader = HttpHandler.Request.ServerVariables["ALL_RAW"];

check out: http://msdn.microsoft.com/en-us/library/ms524602%28VS.90%29.aspx

Answer 12

It might be best to do this outside of your application. You can set up a reverse proxy to do things like this (and much more). A reverse proxy is basically a web server that sits in your server room, and stands between your web server(s) and the client. See http://en.wikipedia.org/wiki/Reverse_proxy