Appengine datastore phantom entity - inconsistent state?

Question

Getting a weird error on java appengine code that used to work fine (nothing has changed but the data in the datastore).

I'm trying to iterate over the results of a query and change a few properties of the entities. The query does return a set of results, however, when I try to access the first result in the list, it throws an exception when trying to access any of its properties (but its key). Here's the exception:

org.datanucleus.state.JDOStateManagerImpl isLoaded: Exception thrown by StateManager.isLoaded
Could not retrieve entity of kind OnTheCan with key OnTheCan(3204258)
org.datanucleus.exceptions.NucleusObjectNotFoundException: Could not retrieve entity of kind OnTheCan with key OnTheCan(3204258)
 at org.datanucleus.store.appengine.DatastoreExceptionTranslator.wrapEntityNotFoundException(DatastoreExceptionTranslator.java:60)

And here is my code:

PersistenceManager pm = PMF.get().getPersistenceManager();
Query query = null;
List<OnTheCan> cans;
query = pm.newQuery("SELECT this FROM " + OnTheCan.class.getName() + " WHERE open == true ORDER BY onTheCanId ASC");
query.setRange(0, num);
cans = (List<OnTheCan>) query.execute();
for (OnTheCan c : cans)
{
 System.err.println(c.getOnTheCanId()); // this works fine! getting the key works
 c.setOpen(false); // failure here with the above exception
 c.setAutoClosed(true);
 c.setEndTime(new Date(c.getStartTime().getTime() + 600000/*10*60*1000*/));
}
pm.close();

The code throws the exception when trying to execute c.setOpen(false) - thats the first time I'm accessing or setting a property that isnt the key. So it seems there is a phantom entity in my datastore with key 3204258. THis entity doesn't really exist (queried the datastore from admin console) but for some reason its being returned by the query. Could my data store be in an inconsistent state?

I've managed the following workaround by placing it as the first line in my for loop. Clearly an ugly hack:

if (c.getOnTheCanId() == 3204258) { continue; }

Any ideas?

Answer 1

I think one of your indexes might have been corrupted. You may need to check http://appengine.google.com/datastore/indexes?&app_id={your_app_id} and see if any of them report an error. If so they may need to be rebuilt.

This might be helpful(taken from http://osdir.com/ml/GoogleAppEngine/2009-05/msg00904.html):

To vacuum and rebuild your indexes:

1. Create a backup of your index.yaml specification.
2. Determine the indexes in state ERROR from your admin console:http://appengine.google.com/datastore/indexes?&app_id={app_id}
3. Remove the definitions of the indexes in ERROR from your index.yaml file.
4. Run "appcfg.py vacuum_indexes your_app_dir/"
5. Wait until the ERROR indexes no longer appear in your admin console.
6. Replace the modified version of your index.yaml file with the original.
7. Run "appcfg.py update_indexes your_app_dir/"

Answer 2

It's probably an index corruption like Andrei said, but if you are unwilling to rebuild all of your indexes, try removing just that entity. Something like this:

  if (c.getOnTheCanId() == 3204258) {
            //continue;
            pm.deletePersistent(c); 
        }

After you can test it again without the if-clause...

---

OK then, there are a few other things that you can try

1. Check your automatic and custom indexes (the index.yaml file). See if you don't use some query anymore but you have it indexed anyway. Maybe this is why you have wrong index table.

2. See if you can query the entity with that id. Like this:

   Key k = KeyFactory.createKey(OnTheCan.class.getSimpleName(),3204258); pm.getObjectById(OnTheCan.class, k);

   and then try to remove it.

3. See if you can get the properties of the entity, maybe it's a different kind that you'd expect.