Adding 90000 XElement to XDocument

Question

I have a Dictionary<int, MyClass>

It contains 100,000 items

10,000 items value is populated whilst 90,000 are null.

I have this code:

var nullitems = MyInfoCollection.Where(x => x.Value == null).ToList();
nullitems.ForEach(x => LogMissedSequenceError(x.Key + 1));

private void LogMissedSequenceError(long SequenceNumber)
        {
            DateTime recordTime = DateTime.Now;

            var errors = MyXDocument.Descendants("ERRORS").FirstOrDefault();
            if (errors != null)
            {

                errors.Add(
                    new XElement("ERROR",
                        new XElement("DATETIME", DateTime.Now.ToString("dd/MM/yyyy HH:mm:ss:fff")),
                        new XElement("DETAIL", "No information was read for expected sequence number " + SequenceNumber),
                        new XAttribute("TYPE", "MISSED"),
                        new XElement("PAGEID", SequenceNumber)
                        )
                );
            }
        }

This seems to take about 2 minutes to complete. I can't seem to find where the bottleneck might be or if this timing sounds about right?

Can anyone see anything to why its taking so long?

Answer 1

This is what I would most likely do.

private void BuildErrorNodes()
{
    const string nodeFormat = @"<ERROR TYPE=""MISSED""><DATETIME>{0}</DATETIME><DETAIL>No information was read for expected sequence number {1}</DETAIL><PAGEID>{1}</PAGEID></ERROR>";

    var sb = new StringBuilder("<ERRORS>");
    foreach (var item in MyInfoCollection)
    {
        if (item.Value == null) 
        {
            sb.AppendFormat(
                nodeFormat,
                DateTime.Now.ToString("dd/MM/yyyy HH:mm:ss:fff"),
                item.Key + 1
            );
        }
    }

    sb.Append("</ERRORS>");

    var errorsNode = MyXDocument.Descendants("ERRORS").FirstOrDefault();
    errorsNode.ReplaceWith(XElement.Parse(sb.ToString()));
}

Answer 2

If your MyInfoCollection is huge, I wouldn't call ToList() on it just so you can use the ForEach extension method. Calling ToList() is going to create and populate a huge list. I'd remove the ToList() call, and make the .ForEach into a for each statement, or write a .ForEach extension method for IEnumerable<T>.

Then profile it and see how long it takes. One other thing to do is remove the find and null check of the ERRORS element. If it's not there, then don't call the for each statement above. That way you null check it one time instead of 90,000 times.

Plus as Michael Stum pointed out, I'd define a string to hold the value DateTime.Now.ToString("dd/MM/yyyy HH:mm:ss:fff"), then reference it or pass it in. Plus, you don't even use this call:

DateTime recordTime = DateTime.Now;

Answer 3

Did you run a profiler? I'd be interested at why you run DateTime.now twice (as it's relatively expensive to call) and how slow it really is when called 90k times...

Answer 4

How about replacing the method call with a LINQ query?

static void Main(string[] args)
{

    var MyInfoCollection = (from key in Enumerable.Range(0, 100000)
                            let value = (MoreRandom() % 10 != 0)
                                                    ? (string)null
                                                    : "H"
                            select new { Value = value, Key = key }
                           ).ToDictionary(k => k.Key, v => v.Value);

    var MyXDocument = new XElement("ROOT",
                                    new XElement("ERRORS")
                                  );
    var sw = Stopwatch.StartNew();
    //===
    var errorTime = DateTime.Now.ToString("dd/MM/yyyy HH:mm:ss:fff");
    var addedIndex = MyInfoCollection.Select((item, index) =>
                                                    new
                                                    {
                                                        Value = item.Value,
                                                        Key = item.Key,
                                                        Index = index
                                                    });
    var errorQuery = from item in addedIndex
                     where string.IsNullOrEmpty(item.Value)
                     let sequenceNumber = item.Key + 1
                     let detail = "No information was read for expected " +
                                  "sequence number " + sequenceNumber
                     select new XElement("ERROR",
                        new XElement("DATETIME", errorTime),
                        new XElement("DETAIL", detail),
                        new XAttribute("TYPE", "MISSED"),
                        new XElement("PAGEID", sequenceNumber)
                        );

    var errors = MyXDocument.Descendants("ERRORS").FirstOrDefault();
    if (errors != null)
        errors.Add(errorQuery);
    //===
    sw.Stop();
    Console.WriteLine(sw.ElapsedMilliseconds); //623
}
static RandomNumberGenerator rand = RandomNumberGenerator.Create();
static int MoreRandom()
{
    var buff = new byte[1];
    rand.GetBytes(buff);
    return buff[0];
}