Ensuring data is in date order using 2 paged lists

Question

I have a 2 streams of data both of which contain date information.

I get 10 items from the first which is a SQL database and 10 items from the second which is a Twitter feed.

When a request comes in to get more data it then retrieves the next 10 items from each source.

The problem I have spotted is that second set of 10 items from the database source may have newer items than the first 10 from Twitter so you have data that looks like this

DB - Today
DB- Today             \\1st result set
Twitter - Yesterday
Twiter - Yesterday
\\\\\\\\\\\\\\\\\\\\\\\\\\\
DB - Today
DB - Today            \\2nd result set
Twitter - 3 days ago 
Twitter - 4 days ago

This obviously doesn't make much sense when a user sees it as the dates are out.

The only thing I can think is to retrieve all the data from twitter rather than 10 records at a time and then using LINQ do a concatenation of database data and twitter data and then order by date.

Can anyone think of a better way?

Answer 1

Wouldn't it be easier to have a data structure that would contain all messages, regardless of their origin?

For example, you could have a list of "Messages" that would be filled by either the database or the Twitter feed. Then you would be able to keep this list ordered by date and display it.

I think this would ease the process.

Answer 2

If you want to blend two sources into a single paged result, you have to fetch extra data. Consider these examples with pagesize = 10

For page 1:

• Request 10 records from DB
• Request 10 records from twitter

Since all 10 records might come from one source, you have to have 10 records from each source ready to go.

---

For page 2 without knowing what's on page 1:

• Request 20 records from DB
• Request 20 records from twitter.

Page 2 might consist of the first 10 records from the source that didn't appear on page 1, or the second 10 records from the source that did appear on page 1... so you must request 40 records to cover everything that could possibly appear on page 2.

---

For page 2 knowing what's on page 1, take the unused page1 records and request enough records to bring you to 10 from each source.

• If page 1 looks like (10 from DB, 0 from twitter), then you only need to request the next 10 records from DB.
• If page 1 looks like (5 from DB, 5 from twitter), then you only need to request the next 5 records from each source.

Once you have 10 new records from each source, solve the problem just like page1.

---

For page 3 and beyond,

• Either request pagenumber*pagesize*2 records from each source
• Or request enough records to bring you to 10 new records from each source to consider for this page.

Answer 3

I went with returning all data from Twitter and merging it into the data from the database:

var dbAllData = Post.All();  //IQueryable

            UserTimelineOptions options = new UserTimelineOptions();
            options.ScreenName = "USERNAME";
            options.Count = 5000;
            options.IncludeRetweets = true;

            TwitterStatusCollection recentTweets = TwitterTimeline.UserTimeline(options);


            var dbAllMerged = dbAllData.AsEnumerable().Select((title) => new BlogData { Title = title.Title, Date = title.Date, RelativeDate = title.Date.ToRelativeDate(), Text = title.Text, DBRecord = true });

            dbAllMerged = dbAllMerged.Concat(recentTweets.Where(y => !y.Text.StartsWith("@")).Select((tweet) => new BlogData { Title = "", Date = tweet.CreatedDate, RelativeDate = tweet.CreatedDate.ToRelativeDate(), Text = tweet.Text, DBRecord = false })).OrderByDescending(x => x.Date);