Puzzling App Engine Datastore Issue (ListProperty)

Question

Ok so I have the same python code locally and in the gae cloud.

when I store an entity locally, the ListProperty field of set element type datetime.datetime looks like so in the Datastore Viewer:

2009-01-01 00:00:00,2010-03-10 00:00:00

when I store same on the cloud, the viewer displays:

[datetime.datetime(2009, 1, 1, 0, 0), datetime.datetime(2010, 3, 9, 0, 0)]

why the different representation?

This wouldn't bother me, only when I query on this field on the cloud the query fails to find the matched entity (it should and it does locally) - leading me to believe it's this differing representation that is causing the trouble. I should repeat - the code is identical.

Anyone think of a reason why this is happening and a solution to it?

UPDATE: my query is as follows (using filters):

from x import y
from datetime import datetime
from google.appengine.ext import db

q = y.EntityType.all().filter('displayDateRange <=',datetime.now()).filter('displayDateRange >=',datetime.now())

usersResult = q.fetch(100)
print `len(usersResult)`

result should be 1, instead it's 0.

Actually it's just the ListProperty with specified value datetime.datetime that is the issue - queries on the StringListProperty is working as expected on the cloud.

I tried the raw filter via interactive console on both local and cloud and cloud gives me no results. So it is a datastore thing, I'm assuming it must have something to do with the storage format - I only have one entity value in both datastores with the ListProperty looking like:

2009-01-01 00:00:00,2010-03-09 00:00:00
[datetime.datetime(2009, 1, 1, 0, 0), datetime.datetime(2010, 3, 9, 0, 0)]

on local and cloud respectively.

Any ideas?

Further Update

Replaced the datetime.now() with hardcoded datetime obj - example filter now looks like:

y.EntityType.all().filter('displayDateRange <=',datetime(2009,11,24)).filter('displayDateRange >=',datetime(2009,11,24))

Note with the above datetime ListProperty range from 1.1.2009 to 3.9.2010 this should return the above entity - I tried this identical filter on localhost dev server and it did so. The cloud, with it's different representation of the datetime.datetime ListProperty, does not.

Note this is taken from the current best practice for filtering on date range

Any ideas what could be wrong?

Answer 1

The problem your see is clearly a conversion to string (calling __str__ or __unicode__) in the local case, while the representation (repr) of your data is displayed on the cloud. But this difference in printing out the results should not be the cause of your failed query on the cloud.

What is your exact query?

UPDATE after knowing the query:

I don't really understand why do you use these filter conditions:

.filter('displayDateRange <=',datetime.now()).filter('displayDateRange >=',datetime.now())

There are two problems with this:

• You call datetime.now() twice, which can give you different results, which would result in an empty result set. It is especially true on a loaded server with multiple threads/processes of execution active at the same time.

• What you might intended to do with the above pair of filters is checking for equality. But it won't work if the precision of the datetime instance returned by datetime.now() and the precision of the datetime stored in the database differs. It is not a good idea to check for equality in the case of floating point numbers and sub-second precision time values in general.

What do you want to achieve with such a pair of filter conditions?

Answer 2

Ok long story short: it's now classed as a bug in the app engine dev server version and is no longer supported in the production cloud datastore.

Filled out a further explanation in a blog post, check out point 3.