Best way to write DataTables from a Web Service into XML?

Question

I'm trying to refactor some slow running code which writes XML using nested loops of several datatables. I read that using linq to write the xml would be faster. I'm not well versed in linq, so I was hoping to get some help here.

Some things I need to mention is that the current architecture uses a webservice which returns data to us in dataTables. We then go through the datatables (iteratively), and there are several which results several nested loops.

example:

dt1 = Webservice.getStuff();

for each (datarow r1 in dt1.Rows) {

   dt2 = Webservice.getMoreStuff(r1[col1], r1[col2]);
   // write out some xml

   for each (datarow r2 in dt2.Rows) {

       dt3 = Webservice.getEvenMoreStuff(r2[col1], r2[col2]);
       // write out more xml 

       for each (datarow r3 in dt3.Rows) {
            // write out more xml 
       }

   }
}

As you can see for obvious reasons, this is terribly slow. Is there a way to speed this up using linq? What would you guys suggest as a more efficient approach to refactor this? I'm sorry if the details are vague...

I appreciate any help anyone could offer.

Answer 1

Writing the XML isn't what is slowing you down in this case. That speed should be negligible compared to the time spent making all of the web service calls.

Instead of focusing on the writing of the XML, I would try to figure out how to compress the number of web service calls so you can get all your data at once rather than making nested calls like that.

Answer 2

I am afraid there is no cure for your need. Because I am pretty sure what makes this method slow is not how you write out xml but the way you acquire the data. If there would be any improvment in writing out xml, it would not be in a noticable proportion. I suggest you to revise the way you acquire data. Try to minimize the number of WebService calls.

Answer 3

It sounds like you need to profile your app first and foremost - maybe a free trial of ANTS or similar will work for you.

Answer 4

Using linq to xml and linq to datasets you can create your own xml like this:

  static void Main(string[] args)
  {

      DataTable t = getStuff("test");

      var xml = new XElement("Main", from row in t.AsEnumerable()
                select new XElement("firstlevel",
                    new XAttribute("a", row["a"]),
                    new XAttribute("b", row["b"]),
                    from row2 in getStuff(row["a"].ToString()).AsEnumerable()
                    select new XElement("secondlevel",
                       new XAttribute("a", row2["a"]),
                       new XAttribute("b", row2["b"]),
                       from row3 in getStuff(row2["a"].ToString()).AsEnumerable()
                       select new XElement("thirdlevel",
                           new XElement("a", row3["a"]),
                           new XElement("b", row3["b"])))));

      Console.WriteLine(xml.ToString());


  }

  private static DataTable getStuff(string s)
  {
      Random r=new Random(s.GetHashCode());
      DataTable t = new DataTable();
      t.Columns.Add("a");
      t.Columns.Add("b");
      for (int i = 0; i < 2; i++)
      {
          t.Rows.Add (r.Next().ToString(), r.Next().ToString());
      }
      return t;
  }

Answer 5

You would have to re factor your solution a bit to get rid of all that looped WS calls. That is a serious performance downer.

Answer 6

You don't say how you are writing the XML or where you are writing the XML to.

If you are writing to a string. Stop that, and write a stream.

If you are writing to a buffered string, then look at the size of the buffer. If you are writing from an ASP.NET page or handler, then call Response.Flush() at regular intervals.

There's a balance here, as writing to a buffer is almost always faster than writing to a stream. However, writing to an automatically resizing buffer will get slower the more it has to resize. What's more, whatever is processing the XML can't begin to act until it starts to receive some, which won't happen until after the first flush.

There could therefore be scope for improvement here, though the calls to the webservice are likely to outweigh what gains can be made. Maybe this can be improved too, if you rewrite the parsing of the webservice's response so that it yields items as they are parsed, which means that you can start processing the response before the entire response is received.