What in an HTML page is unique?

Question

My question is about verification more than anything else. What can be used to determine what is unique in an HTML document? (The document can have a degree of being dynamic.)

What is able to be used, or generated to recognize that a page is the correct page to an accuracy of say 99%, taking into consideration you can store a "fingerprint" of sorts of the page you are verifying?

---

For clarity, this is an added extra to encryption/https etc. This page can and will change with dynamic content according to specific users, however so can the fingerprint, but a single fingerprint cannot 100% match 100% of users due to the nature of dynamic content. Therefore a hash cannot work here, at least not in a simplistic form.

Answer 1

You can't be even 1% sure if you won't check IP of host. The next is encryption. (Without this you can be a victim of ARP poisoing (only in lan networks)).

The key in HTTPS has to be the same all the time.

If it changes it means that someone is cheating or the key got update (the keys have their expiration date.)

Answer 2

The fingerprint of the page is the host-name,port,and path. That is the only thing guaranteed to be unique across the web. I suppose you could also include the cache headers (Last-Modified) to see if it changed.

On top of this if you hashed the html you could see if it changed even if last-modified header changed.

Answer 3

Assuming for a minute that you want to store a 'fingerprint' of a HTML page so you can recognise it later if it exactly matches, just use a simple hash digest of the HTML page.

Unless you clearify the question more, I can see no reason of why it should matter that it is HTML or what browser it is in.

This won't tell you if the page is at the same location however. For that you would need to store additional details such as host/ip and path.

Answer 4

A unique fingerprint of a HTML page is easy to calculate. Build a hash from the following:

• protocol: http or https
• URL: domain + uri
• Query_string
• the exact page's contents down to a byte

Optionally some headers:

• Server
• Content-Type this is important
• Content-encoding this probably too
• more ideas? Feel free to edit them in.

this assumes you're not POSTing any data to pages.

Answer 5

If you can get the text versions of the two pages you could diff them. You could determine a maximum range acceptable for differences in the page.

There is a Unix util (called diff). There are win32 versions of this tool floating around the net also. Wikipedia has an article on diff: http://en.wikipedia.org/wiki/Diff.

The wiki article lists free file comparison tools and the "See also" section has links to other articles that discuss file comparison tools and delta encoding.

The "Levenshtein distance metric" may also be an interesting approach.

There is a decent C# Difference engine on CodeProject. I can't post another link due to my low points but the article title is: "A Generic, Reusable Diff Algorithm in C#".

Answer 6

Even if you had the exact hostname, port, and path the content still could be different if there is an app server serving the web pages or if the web server is inserting ad content.

If you could reliably identify the parts of the HTML that are dynamic (like ads or timestamps that keep updating) then I would normalize the data first. I'd strip out all space characters (spaces, tabs, newlines) then make a hash of that content.

I would not include the hostname-port-path in the hash because that wouldn't add anything to the "fingerprint". (That info is useful later when you have to requery the web server later to compare the HTML.)