Limit calls to external database with Python CGI

Question

I've got a Python CGI script that pulls data from a GPS service; I'd like this information to be updated on the webpage about once every 10s (the max allowed by the GPS service's TOS). But there could be, say, 100 users viewing the webpage at once, all calling the script.

I think the users' scripts need to grab data from a buffer page that itself only upates once every ten seconds. How can I make this buffer page auto-update if there's no one directly viewing the content (and not accessing the CGI)? Are there better ways to accomplish this?

Answer 1

Cache the results of your GPS data query in a file or database (sqlite) along with a datetime.

You can then do a datetime check against the last cached datetime to initiate another GPS data query.

You'll probably run into concurrency issues with cgi and the datetime check though...

To get around concurrency issues, you can use sqlite, and put the write in a try/except. Here's a sample cache implementation using sqlite.

import datetime
import sqlite3 

class GpsCache(object):
    db_path = 'gps_cache.db'
    def __init__(self):
        self.con = sqlite3.connect(self.db_path)
        self.cur = self.con.cursor()

    def _get_period(self, dt=None):
        '''normalize time to 15 minute periods'''
        if dt.minute < 15:
           minute_period = 0
        elif 15 <= dt.minute < 30:
           minute_period = 15
        elif 30 <= dt_minute < 45: 
           minute_period = 30
        elif 45 <= dt_minute:
           minute_period = 25
        period_dt = datetime.datetime(year=dt.year, month=dt.month, day=dt.day, hour=dt.hour, minute=minute_period)
        return period_dt

    def get_cache(dt=None):
        period_dt = self._get_period(dt)
        select_sql = 'SELECT * FROM GPS_CACHE WHERE date_time = "%s";' % period_dt.strftime('%Y-%m-%d %H:%M')
        self.cur.execut(select_sql)
        result = self.cur.fetchone()[0]
        return result


    def put_cache(dt=None, data=None):
        period_dt = self._get_period(dt)
        insert_sql = 'INSERT ....'  # edit to your table structure
        try:
            self.cur.execute(insert_sql)
            self.con.commit()
        except sqlite3.OperationalError:
            # assume db is being updated by another process with the current resutls and ignore
            pass

So we have the cache tool now the implementation side.

You'll want to check the cache first then if it's not 'fresh' (doens't return anything), go grab the data using your current method. Then cache the data you grabbed. you should probably organize this better, but you should get the general idea here.

Using this sample, you just replace your current calls to 'remote_get_gps_data' with 'get_gps_data'.

from gps_cacher import GpsCache

def remote_get_gps_data():
    # your function here
    return data

def get_gps_data():
    data = None
    gps_cache = GpsCache()
    current_dt = datetime.datetime.now()
    cached_data = gps_cache.get_cache(current_dt)    
    if cached_data:
        data = cached_data
    else:
        data = remote_get_gps_data()
        gps_cache.put_cache(current_dt, data)
    return data