libxml2 HTML chunk parsing

Question

I'm downloading HTML from a website. The file can be quite large so while the file's downloading, I want to already parse the available chunks of HTML so that the process appears faster for the end-user of my program. I don't have control over how the cunks are generated, so a chunk can begin in the middle of a word, e.g. like so:

chunk 1 --->  <div class="storyti
chunk 2 --->  tle"><a href="htt
chunk 3 --->  p://www.xkcd.com/">XKCD</a>
...and so on.

I have seen example where libxml2 was used to parse XML chunks exactly how I described. Can libxml2 also parse HTML chunks? I have checked with tidy on the html files I'm going to be downloading, it reports warnings but no errors. Can libxml2 parse those HTML chunks as well?

Answer 1

If the content is XHTML, you can use libxml2 to parse it (since it is in fact XML). If it's regular HTML on the other hand, you'd have to use an SGML parser instead.

Answer 2

Well, it seems it's not valid XHTML. Is there maybe some way to tidy HTML chunks?

Answer 3

libxml2 has a html parser which supports malformed/broken html. Please check the link here.