Appengine Python Bulk Export error

Question

This seems to run for a few thousand records then dies on its arse

import datetime
import time

from google.appengine.ext import db
from google.appengine.tools import bulkloader
from google.appengine.api import datastore_types

class SearchRec(db.Model):
  WebSite = db.StringProperty()
  WebPage = db.StringProperty()
  DateStamp = db.DateTimeProperty(auto_now_add=True)
  IP = db.StringProperty()
  UserAgent = db.StringProperty()


class TrackerExporter(bulkloader.Exporter):
    def __init__(self):
        bulkloader.Exporter.__init__(self, 'SearchRec',
                                    [('WebSite', str, None),
                                    ('WebPage', str, None),
                                    ('DateStamp', lambda x: str(datetime.datetime.strptime(x, '%d/%m/%Y').date()), None),
                                    ('IP', str, None)
                                    ])

exporters = [TrackerExporter]

if __name__ == '__main__':
    bulkload.main(TrackerExporter)

Error>>

  File "tracker-export.py", line 89, in <lambda>
    ('DateStamp', lambda x: str(datetime.datetime.strptime(x, '%d/%m/%Y').date()
), None),
TypeError: strptime() argument 1 must be string, not datetime.datetime

The errors going to drive me mental as (counting the dots) its gone through 42200 records before falling over...

Help!

Answer 1

I'm not sure why exactly that would be happening. I'm not too familiar with the Bulk Exporting facilities of App Engine, but it sounds like the DateStamp field is being given to the bulk exporter as a string (which is what your converter expects) for the first 42200 records and then, for some reason, it is given as a real datetime.datetime object.

Anyway, here's a treatment for this particular symptom:

lambda x: str((x if isinstance(x, datetime.datetime) else datetime.datetime.strptime(x, '%d/%m/%Y')).date())

That should do the right thing if given either a string or a datetime.datetime object. This might be a case where you should give it a dedicated function instead of torturing a lambda like that.