I was running into a similar issue when trying to export to XLS and PDF, the only thing that seem to improve the response time was sending the response directly from the class that generates the file like:
HttpContext.Current.Response.Clear();
HttpContext.Current.Response.ClearContent();
HttpContext.Current.Response.ClearHeaders();
HttpContext.Current.Response.Buffer = true;
HttpContext.Current.Response.BufferOutput = true;
HttpContext.Current.Response.ContentType = "application/pdf";
HttpContext.Current.Response.AddHeader("Content-Disposition", "attachment;filename=" + file + ".pdf");
HttpContext.Current.Response.BinaryWrite(stream.ToArray());
HttpContext.Current.Response.Flush();
stream.Close();
HttpContext.Current.Response.End();
But if you do this you will get a "not all code paths return a value"
from the ActionMethod, to avoid that we just send a:
return new EmptyResult();
This last line will actually never be executed because we end the request directly on the method.