What's the best way to get the data I want from Twitter?

Question

I'm currently saving some Twitter data in MySQL. My host only lets me run cron jobs every hour, so to semi-simulate realtime results, I've copied the same file 6 times, and run one every 10 minutes (the host DOES let you control the hourly offset). This is stupid, I think.

Is there some mechanism I can learn about that would push the data my way? Any thoughts or suggestions welcome.

(I've steered myself away from just querying their server with each page view; I know enough to know that's poor practice)

Answer 1

How about accessing a web page (which will in turn execute the program) hosted at the server by adding to cron at client side (home system):

/usr/bin/curl http://yourserver.com/twitter

Otherwise, you can run the following bash script every hour:

#!/bin/bash

for (( i = 0; i < 6; i += 1 )); do
    /usr/bin/curl 'http://yourserver.com/twitter'
    sleep 600
done

Answer 2

A relatively simple solution is to run a cron job on another computer. It would do the requests to Twitter then execute a HTTP POST to a designated page on the server (e.g. http://foo.com/latestTwitterData). Of course, you would want to have authentication to prevent random crap getting sent to you.

I don't know if this is reasonable for your situation.

Answer 3

It's pretty easy to run code every second or so.

// pseudocode
while(1) {
    // do request

    // sleep 1 second
    sleep(1);
}

Answer 4

Why not just put a while loop into your program and then sleep N seconds between however long you need the updates? You can then die after 59 minutes 30 seconds.

Alternatively, to optimize the copying of multiple files, you can add multiple calls to your program within the single cron line. Something like:

./prog.pl; sleep 60; ./prog.pl

Answer 5

You can sanely pull twitter data triggered from your requests. It's a little esoteric, but essentially you store locking data in a table to ensure only one request polls the data from twitter every N minutes (or whenever you need it). Example:

1. Request checks to see if new twitter data needs to be retrieved
2. Check lock table to see if another request is already talking to twitter
3. Add record to lock table. Make sure to specify data in a column that is set to unique via a database constraint. This will keep you from making two locks.
4. Talk to twitter, save twitter data.
5. Remove lock record

For speed, ensure your lock table is in memory or use memcached instead. Of course, if you can use memcached you probably have full control over cron anyway. :)