Read only when the XML data is updated

Question

Hello guys,

I'm able to parse RSS with PHP - What I'm looking for is to be able to get only the updated content, and do nothing if there's no new update to the RSS.

For example, I have this RSS File, and if there's no new content, nothing happens, but if there's a new content, I want to send my users the latest RSS update, and not resend what they already have. I'm parsing and sending the title and link only.

I use cronjob to check every hour for update. My question is how can I tell that the feed is now updated and not the same as the last one? Here's the PHP file that I'm using to read the RSS. Do I write the last content to file and compare them or is there any other way to determine that the content is now different from the last?

Update: I had to resurrect this post because I'm still trying to get it to work. Although I accepted a few answers, they have been very hard to implement, for example the hashing option looked like a good idea initially, but as thousands of RSS would be checked, it would be almost impossible to hash them all.

Again, someone suggested HTTP Cache - I couldn't find a simple demo so I'm practically stuck.

Any further suggest would be highly appreciated.

Answer 1

Your clients will always be asking for your feed data so you cannot necessarily control when they ask. I dont think most feed readers obey HTTP Cache Control / Expires headers so you cannot rely on using the HTTP spec and leverage HTTP caching.

I think your best bet is to just cache your last response and send all subsequent requests from the cache - updating the cache appropriately when changes are made. Effectively this means that your cost to respond to each client and its stale data is pretty much close to 0, if you just pull it from memcache or the filesystem.

Answer 2

Because of the diversity of rss there is no easy solution for the problem your raised. The main issue is how to determine the uniqueness of the rss item. It can be guid, publish time or content itself, but it maybe tricky to detect that automatically.

Once you know the uniqueness criteria you can persist all 'old' items and compare them to the newest ones you receive.

HTTP Cache Control and Expires headers could be used as an optimization for the sites that support that, but unfortunately some doesn't.

Answer 3

HTTP Conditional GET is probably as close as you're going to get to what you want.

Answer 4

You could use hashes for this, in two ways:

1. To ease updating - When requesting an update, you hash the whole feed and compare the result with the hash from the last time - if they are identical, you know that the feed did not change and can stop before even parsing it.
2. To identify changes - On parsing, you hash each item and compare it to the hashes stored from previous runs. If it matches one, you know that you've seen it before.

If the feed in question offers guids for its items you could refine this process by storing guid<>hash pairs. This would make the comparison quicker, as you would only compare items to known previous versions instead of comparing to all previous items.

You'd still need some expiration/purge mechanism to keep the amount of stored hashes within bounds, but given that you only store relatively short strings (depending on the chosen hash algorithm), you should be able to keep quite a backlog before getting performance problems.

Answer 5

@Henrik's solution is correct, however it might be easiest to supply you with an example of the hashing data:

// hash the three channel variables
$hash = sha1($channel_title . $channel_link . $channel_desc);

// here you should check the currently stored database hashed 
// value against current hash value to see if any channel variables
// have recently changed
if ($database_hash != $hash) {
    // you need to update the channel data in your database
    // including the new hash value
}

for ($i = 0; $i < 3; $i++) {

    // hash the item values
    $hash = $item_title . $item_link . $item_description

    // here you should check the currently stored database hashed 
    // value against all item hash values to see if any item variables
    // have recently changed
    if ($database_hash != $hash) {
        // you need to update the item data in your database
        // including the new hash value
    }

}

Also, if you want to do a quick check to determine if any data in the XML file has changed whatsoever, you could hash the XML as a string. You should store this value and check against it every time you run your cronjob to see if the value has changed (indicating some data within the XML file has changed).

$overall_hash = sha1($xmlDoc->saveXML());