C# Is there a LINQ to HTML, or some other good .Net HTML manipulation API?

Question

Hello all,

I have a C# WPF application that needs to consume data that is exposed on a webpage as a HTML table.

After getting inspiration from this url I tried using Linq to Xml to parse the Html document, but this only works if the HTML document is extremely well formed (and doesn't have any comments or HTML entities inside it). I have managed to get a working solution using this technique, but it is far from ideal.

I am after a solution that is intended for parsing HTML. I have hacked "solutions" before, but they are brittle. I am after a robust way of parsing/manipulating the document. I'd ideally like something that makes the task as easy as it would be from Javascript/JQuery.

Does anyone know of a good .Net library or utility for parsing/manipulating HTML?

Thanks for any help you can offer.

Answer 1

HTML is rarely well-formed enough that you could reliably use LINQ to XML. It's conceivable that you might find an HTML "cleaner" that could fix the formatting well enough to be read, but there's not telling how robust it would be.

I assume this is a "screenscraper" that reads from an HTML table over which you have no control. Don't stress over robustness in this case, screen-scraping is inherently brittle. If your requirements are set in stone, design the scraper to be easily updatable if/when the HTML you are scraping changes.

Answer 2

I had to do this in a recent project and I used LINQ to XML. If you know it's always going to be clean XHTML then you can probably recursively copy the DOM pretty easily, but I used the DevComponents HTMLDocument class library (http://www.devcomponents.com/htmldoc/) to convert HTML to XML then pulled that into an XElement. This reduces the challenge to getting your HTML into an XElement hierarchy. The one caveat is it chokes on script elements, so I deleted those by brute force.

    /// <summary>
    /// Extracts an HtmlDocument DOM to an XElement DOM that can be queried using LINQ to XML.
    /// </summary>
    /// <param name="htmlDocument">HtmlDocument containing DOM of page to extract.</param>
    /// <returns>HTML content as <see cref="XElement" /> for consumption by LINQ to XML.</returns>
    public XElement ExtractXml(HtmlDocument htmlDocument) {
        XmlDocument xmlDoc = htmlDocument.ToXMLDocument();

        // Find and remove all script tags from XML DOM or LINQ to XML will choke on XElement.Parse(XmlDocument).
        IList<XmlNode> nodes = new List<XmlNode>();
        foreach (XmlNode node in xmlDoc.GetElementsByTagName("script"))
            nodes.Add(node);
        foreach (XmlNode node in nodes)
            node.ParentNode.RemoveChild(node);

        return XElement.Parse(xmlDoc.OuterXml);
    }

Answer 3

Even though it's not LINQ based, I suggest researching the HTML Agility Pack from CodePlex.

Note: Html Agility Pack now supports Linq to Objects (via a LINQ to Xml Like interface)

From the HTML Agility Pack page:

This is an agile HTML parser that builds a read/write DOM and supports plain XPATH or XSLT (you actually don't HAVE to understand XPATH nor XSLT to use it, don't worry...). It is a .NET code library that allows you to parse "out of the web" HTML files. The parser is very tolerant with "real world" malformed HTML. The object model is very similar to what proposes System.Xml, but for HTML documents (or streams).

Answer 4

I've posted some code providing "LINQ to HTML" functionality here:

http://stackoverflow.com/questions/100358/looking-for-c-html-parser/624410#624410

Answer 5

There's a LINQ to HTML library here:

http://www.justagile.com/linq-to-html.aspx