Import rss feeds and usability

Question

Hi all,

I run a small blog network and on this I have a page where I show the latest blog posts from different blogs on my server. I would like to extend this page, to also include new posts from external blogs using rss feeds.

Currently it’s easy to get the content, since it’s just a simple query selecting posts by date, but it troubles me to see how to make the most effective design when extending it.

The easiest solution would be to periodic run a cronjob that import posts from the external sites, and then save them in the database. Though this creates the possibility that the posts could be altered in content or removed by the author, leaving me to display ”invalid content”.

The best solution would be if I don’t have to save the posts, and instead just import them directly on the page. But how would this affect usability and loading time? Is it somehow possible to cache the feeds? If I should choose a combination of displaying internal and external posts using a query and importing feeds directly, how can this be combined to use ”pagination” (10 results pr. page)?

I hope someone can help me with a small proof of concept code, or describe what they believe would be the most effective way of handling this.

PS: For importing feeds I use SimplePie http://simplepie.org

Thanks in advance

Answer 1

If you already use SimplePie then you can use its caching mechanism to have the feed data cached.

To combine the articles from internal and external sources create a data structure with all articles. This can be an array of all items sorted by publication timestamp. Then from this array choose the articles for a certain page number.

Here's some code to create a combined array of posts. This should give you a idea of the steps involved. The Post class represents a post. The internal and external posts are converted to a Post and stored in the array $posts. This array is sorted by timestamp and at the end all posts are echoed.

$internalPosts must contain the posts form your system and $feedUrls the URL's of the external feeds. Since I don't know the structure of the internal posts you must adapt the part where internal posts are converted to generic posts.

$internalPosts = array();
$feedUrls = array();

include_once 'simplepie.inc';

class Post {
    public $title;
    public $link;
    public $description;
    public $publishedAt;

    public function __construct($title, $link, $description, $publishedAt) {
        $this->title = $title;
        $this->link = $link;
        $this->description = $description;
        $this->publishedAt = $publishedAt;
    }   
}

$posts = array();

// Convert internal posts to generic post.
foreach($internalPosts as $item){
    $posts[] = new Post($item->title, $item->link, $item->description, $item->publishedAt);
}

// Retrieve feeds and add posts.
$feed = new SimplePie();

foreach($feedUrls as $url){
    $feed->set_feed_url($url);
    $feed->init();

    foreach ($feed->get_items() as $item) {
        $posts[] = new Post($item->get_title(), $item->get_link(), $item->get_description(), $item->get_date('U'));
    }
}

// Sort function.
function byPublicationTimestamp($itemA, $itemB){
    return ($itemB->publishedAt - $itemA->publishedAt);
}

usort($posts, 'byPublicationTimestamp');

foreach($posts as $post){
    echo "<p><a href='$post->link'>$post->title</a><br/>" . date('l, j F Y', $post->publishedAt) . " - $post->description</p>"; 
}

For improved performance consider storing the combined articles separately and build the pages from this data. Then you need to update this combined data anytime a new article is published internally or the cached version of an external feed has been refreshed.

If you need to publish the external content shortly after it's published on the original site then I would contact those sites to see if it's possible to get a notification of updates instead of waiting for the cached version to expire.

EDIT: added sample code.