Suggestions on how build an HTML Diff tool?

Question

In this post I asked if there were any tools that compare the structure (not actual content) of 2 HTML pages. I ask because I receive HTML templates from our designers, and frequently miss minor formatting changes in my implementation. I then waste a few hours of designer time sifting through my pages to find my mistakes.

The thread offered some good suggestions, but there was nothing that fit the bill. "Fine, then", thought I, "I'll just crank one out myself. I'm a halfway-decent developer, right?".

Well, once I started to think about it, I couldn't quite figure out how to go about it. I can crank out a data-driven website easily enough, or do a CMS implementation, or throw documents in and out of BizTalk all day. Can't begin to figure out how to compare HTML docs.

Well, sure, I have to read the DOM, and iterate through the nodes. I have to map the structure to some data structure (how??), and then compare them (how??). It's a development task like none I've ever attempted.

So now that I've identified a weakness in my knowledge, I'm even more challenged to figure this out. Any suggestions on how to get started?

clarification: the actual content isn't what I want to compare -- the creative guys fill their pages with lorem ipsum, and I use real content. Instead, I want to compare structure:

<div class="foo">lorem ipsum<div>

is different that

<div class="foo">
<p>lorem ipsum<p>
<div>

Answer 1

Open each page in the browser and save them as .htm files. Compare the two using windiff.

Answer 2

@Mike - that would compare everything, including the content of the page, which isn't want the original poster wanted.

Assuming that you have access to the browser's DOM (by writing a Firefox/IE plugin or whatever), I would probably put all of the HTML elements into a tree, then compare the two trees. If the tag name is different, then the node is different. You might want to stop enumerating at a certain point (you probably don't care about span, bold, italic, etc. - maybe only worry about divs?), since some tags are really the content, rather than the structure, of the page.

Answer 3

The DOM is a data structure - it's a tree.

Answer 4

See this previous post and accompanying answers.

Answer 5

I don't know any tool but I know there is a simple way to do this:

• First, use a Regular Expression tool to strip off all the text in your HTML file. You can use this regular expression to search for the text (?<=^|>)[^><]+?(?=<|$) and replace them by an empty string (""), ie delete all the text. After this step, you will have all HTML markup tags. There are a lot of free regular expression tool out there.
• Then, you repeat the first step for the original HTML file.
• Last, you use a diff tool to compare the two sets of HTML markups. This will show what is missing between one set and the other.

Answer 6

Run both files through the following Perl script, then use diff -iw to do a case-insensitive, whitespace-ignoring diff.

#! /usr/bin/perl -w

use strict;

undef $/;

my $html = <STDIN>;

while ($html =~ /\S/) {
  if ($html =~ s/^\s*<//) {
    $html =~ s/^(.*?)>// or die "malformed HTML";
    print "<$1>\n";
  } else {
    $html =~ s/^([^<]+)//;
    print "(text)\n";
  }
}

Answer 7

This has been an excellent start. A few more clarifications/comments:

• I probably don't care about IDs, since .net will mangle them
• some of the structure will be in a repeater or other such control, so I might end up having more or fewer repeating elements

further thought: I think a good start would be to assume the html is XHTML compliant. I could then infer the schema (using the new .net XmlSchemaInference methods), then diff the schemata. I can then look at the differences and consider whether or not they're significant.

Answer 8

If i was to tacke this issue I would do this:

1. Plan for some kind of a DOM for html pages. starts at lightweight and then add more as needed. I would use composite pattern for the data structure. i.e. every element has children collection of the base class type.
2. Create a parser to parse html pages.
3. Using the parser load html element to the DOM.
4. After the pages' been loaded up to the DOM, you have the hierachical snapshot of your html pages structure.
5. Keep iterating through every element on both sides till the end of the DOM. You'll find the diff in the structure, when you hit a mismatched of element type.

In your example you would have only a div element object loaded on one side, on the other side you would have a div element object loaded with 1 child element of type paragraph element. fire up your iterator, first you'll match up the div element, second iterator you'll match up paragraph with nothing. You've got your structural difference.

Answer 9

I think some of the suggestions above don't take into account that there are other tags in the HTML between two pages which would be textually different, but the resulting HTML markup is functionally equivalent. Danimal lists control IDs as an example.

The following two markups are functionlly identical, but would show up as different if you simply compared tags:

<div id="ctl00_TopNavHome_DivHeader" class="header4">foo</div>
<div class="header4">foo</div>

I was going to suggest Danimal write an HTML translation which looks for the HTML tags and converts both docs into a simplified version of both which omits ID tags and any other tags you designate as irrelevant. This’d likely have to be a work in progress, as you ignore certain attributes/tags and then run into new ones which you also want to ignore.

However, I like the idea of using the XmlSchemaInterface to boil it down to the XML schema, then use a diff tool which understands XML rules.

Answer 10

My suggestion is just the basic way of doing it... Of course to tackle the issue you mentioned additional rules must be applied here... Which is in your case, we got a matching div element, and then apply attributes/property matching rules and what not...

To be honest, there are many and complicated rules that need to be applied for the comparison, and its not just a simple matching element to another element. For example what happens if you have duplicates. e.g. 1 div element on one side, and 2 div element on the other side. How are you gonna match up which div elements matches together?

There are alot other complicated issues that you will find in the comparison word. Im speaking based of experience (part of my job is to maitain my company text comparison engine).

Answer 11

See http://www.semdesigns.com/Products/SmartDifferencer/index.html for a tool that is parameterized by langauge grammar, and produces deltas in terms of language elements (identifiers, expressions, statements, blocks, methods, ...) inserted, deleted, moved, replaced, or has identifiers substituted across it consistently. This tool ignores whitespace reformatting (e.g., different linebreaks or layouts) and semantically indistinguishable values (e.g., it knows that 0x0F and 15 are the same value). This can be applied to HTML using an HTML parser.

EDIT: 9/12/2009. We've built an experimental SmartDiff tool using an HTML editor.

Answer 12

If i were to do this, first i would learn HTML. (^-^) Then i would build a tool that strips out all of the actual content and then saves that as a file so it can be piped through WinDiff (or other merge tool).

Answer 13

Take a look at beyond compare. It has an XML comparison feature that can help you out.

Answer 14

You may also have to consider that the 'content' itself could contain additional mark-up so it's probably worth stripping out everything within certain elements (like <div>s with certain IDs or classes) before you do your comparison. For example:

<div id="mainContent">
<p>lorem ipsum etc..</p>
</div>

and

<div id="mainContent">
<p>Here is some real content<img class="someImage" src="someImage.jpg" /></p>
<ul>
<li>and</li>
<li>some</li>
<li>more..</li>
</ul>
</div>

Answer 15

I would use (or contribute to) html5lib and its SAX output. Just zip through the 2 SAX streams looking for mismatches, and highlight the whole corresponding subtree (would be hella confusing though).

Answer 16

http://www.mugo.ca/Products/Dom-Diff

Works with FF 3.5. I haven't tested FF 3.6 yet.