how to determine if webpage has been modified

Question

hello,

I have snapshots of multiple webpages taken at 2 times. What is a reliable method to determine which webpages have been modified?

I can't rely on something like an RSS feed, and I need to ignore minor noise like date text.

Ideally I am looking for a Python solution, but an intuitive algorithm would also be great.

Thanks!

Answer 1

Something like Levenshtein Distance could come in handy if you set the threshold of the changes to a distance that ignored the right amount of noise for you.

Answer 2

Well, first you need to decide what is noise and what isn't. You can use a HTML parser like BeautifulSoup to remove the noise, pretty-print the result, and compare it as a string.

If you are looking for an automatic solution, you can use difflib.SequenceMatcher to calculate the differences between the pages, calculate the similarity and compare it to a threshold.

Answer 3

The solution really depends if you are scraping a specific site, or are trying to create a program which will work for any site.

You can see which areas change frequently doing something like this:

 diff <(curl http://stackoverflow.com/questions/) <(sleep 15; curl http://stackoverflow.com/questions/)

If your only worried about a single site, you can create some sed expressions to filter out stuff like time stamps. You can repeat until no difference is shown for small fields.

The general problem is much harder, and I would suggest comparing the total word count on a page for starters.

Answer 4

just take snapshots of the files with MD5 or SHA1...if the values differ the next time you check, then they are modified.