Using Google App Engine's Cron service to extract data from a URL

Question

Hi Guys!

I need to scrape a simple webpage which has the following text:

Value=29 Time=128769

The values change frequently.

I want to extract the Value (29 in this case) and store it in a database. I want to scrape this page every 6 hours. I am not interested in displaying the value anywhere, I just am interested in the cron. Hope I made sense.

Please advise me if I can accomplish this using Google's App Engine.

Thank you!

Answer 1

Please advise me if I can accomplish this using Google's App Engine.

Sure! E.g., in Python, urlfetch (with the URL as argument) to get the contents, then a simple re.search(r'Value=(\d+)').group(1) (if the contents are as simple as you're showing) to get the value, and a db.put to store it. Do you want the Python details spelled out, or do you prefer Java?

Edit: urllib / urllib2 would also be feasible (GAE does support them now).

So cron.yaml should be something like:

cron:
- description: refresh "value"
  url: /refvalue
  schedule: every 6 hours

and app.yaml something like:

application: valueref
version: 1
runtime: python
api_version: 1

handlers:
- url: /refvalue
  script: refvalue.py
  login: admin

You may have other entries in either or both, of course, but this is the subset needed to "refresh the value". A possible refvalue.py might be:

import re
import wsgiref.handlers

from google.appengine.ext import db
from google.appengine.ext import webapp
from google.appengine.api import urlfetch

class Value(db.Model):
  thevalue = db.IntegerProperty()
  when = db.DateTimeProperty(auto_now_add=True)

class RefValueHandler(webapp.RequestHandler):
  def get(self):
    resp = urlfetch.fetch('http://whatever.example.com')
    mo = re.match(r'Value=(\d+)', resp.content)
    if mo:
      val = int(mo.group(1))
    else:
      val = None
    valobj = Value(thevalue=val)
    valobj.put()

def main():
  application = webapp.WSGIApplication(
    [('/refvalue', RefValueHandler),], debug=True)
  wsgiref.handlers.CGIHandler().run(application)

if __name__ == '__main__':
  main()

Depending on what else your web app is doing, you'll probably want to move the class Value to a separate file (e.g. models.py with other models) which of course you'll then have to import (from this .py file and from others which do something interesting with all of your saved values). Here I've taken some possible anomalies into account (no Value= found on the target page) but not others (the target page's server does not respond or gives an error); it's hard to know exactly what anomalies you need to consider and what you want to do if they occur (what I'm doing here is very simply recording None as the value at the anomaly's time, but you may want to do more... or less -- I'll leave that up to you!-)