How can I use bulkuploader to populate class with a db.SelfReferenceProperty?

Question

I've got a class that is using db.SelfReferenceProperty to create a tree-like structure.

When trying to populate the database using appcfg.py upload_data -- config_file=bulkloader.yaml --kind=Group --filename=group.csv (...) , I'm getting an exception saying BadValueError: name must not be empty. ^{(Full stack below)}

I tried ordering the data to make sure that a Groups that had a foreign key pointing at them were first. That didn't work.

By commenting from the bulkloader.yaml the line making the transformation "import_transform: transform.create_foreign_key('Group')", the data is uploaded, but it stores that property as string, breaking my application logic.

- kind: Group
  connector: csv
  connector_options:
  property_map:
    - property: __key__
      external_name: key
      export_transform: transform.key_id_or_name_as_string

    - property: name
      external_name: name
      # Type: String Stats: 9 properties of this type in this kind.

    - property: section
      external_name: section
      # Type: Key Stats: 6 properties of this type in this kind.
      import_transform: transform.create_foreign_key('Group')
      export_transform: transform.key_id_or_name_as_string

Is there any way to make the bulkloader take into account selfreferences, or should I either make a transformation serverside of the bulkloaded data or implement my own bulk loading algorithm?

----
Traceback (most recent call last):
  File "/home/username/src/google_appengine/google/appengine/tools/adaptive_thread_pool.py", line 150, in WorkOnItems
    status, instruction = item.PerformWork(self.__thread_pool)
  File "/home/username/src/google_appengine/google/appengine/tools/bulkloader.py", line 691, in PerformWork
    transfer_time = self._TransferItem(thread_pool)
  File "/home/username/src/google_appengine/google/appengine/tools/bulkloader.py", line 846, in _TransferItem
    self.content = self.request_manager.EncodeContent(self.rows)
  File "/home/username/src/google_appengine/google/appengine/tools/bulkloader.py", line 1267, in EncodeContent
    entity = loader.create_entity(values, key_name=key, parent=parent)
  File "/home/username/src/google_appengine/google/appengine/ext/bulkload/bulkloader_config.py", line 382, in create_entity
    return self.dict_to_entity(input_dict, self.bulkload_state)
  File "/home/username/src/google_appengine/google/appengine/ext/bulkload/bulkloader_config.py", line 133, in dict_to_entity
    self.__run_import_transforms(input_dict, instance, bulkload_state_copy)
  File "/home/username/src/google_appengine/google/appengine/ext/bulkload/bulkloader_config.py", line 230, in __run_import_transforms
    value = self.__dict_to_prop(transform, input_dict, bulkload_state)
  File "/home/username/src/google_appengine/google/appengine/ext/bulkload/bulkloader_config.py", line 188, in __dict_to_prop
    value = transform.import_transform(value)
  File "/home/username/src/google_appengine/google/appengine/ext/bulkload/bulkloader_parser.py", line 93, in __call__
    return self.method(*args, **kwargs)
  File "/home/username/src/google_appengine/google/appengine/ext/bulkload/transform.py", line 114, in generate_foreign_key_lambda
        return datastore.Key.from_path(kind, value)
  File "/home/username/src/google_appengine/google/appengine/api/datastore_types.py", line 384, in from_path
    ValidateString(id_or_name, 'name')
  File "/home/username/src/google_appengine/google/appengine/api/datastore_types.py", line 109, in ValidateString
    raise exception('%s must not be empty.' % name)
BadValueError: name must not be empty.

Answer 1

Using an answer given to a similar problem as a base, I could successfully solve this by creating a small helpers.py file to do act as a wrapper around transform.create_foreign_key:

from google.appengine.api import datastore

def create_foreign_key(kind, key_is_id=False):
  def generate_foreign_key_lambda(value):
    if value is None:
      return None

    if key_is_id:
      value = int(value)
    try:
      return datastore.Key.from_path(kind, value)
    except:
      return None

  return generate_foreign_key_lambda

Once that file is in place in the same directory as your yaml bolk upload configuration file (bulkloader.yaml), you add this to that file:

python_preamble:
- (...)
- import: helpers

transformers:

- kind: Group
  connector: csv
  connector_options:
  property_map:
    - property: __key__
      external_name: key
      export_transform: transform.key_id_or_name_as_string

    - property: name
      external_name: name

    - property: section
      external_name: section
      import_transform: helpers.create_foreign_key('Group')
                      # ^^^^^^^ we use the wrapper instead
      export_transform: transform.key_id_or_name_as_string

With those changes, the bulk upload is now working correctly.

Before using this, you should definitely modify the catch all except, and replace it with may be except BadValueError.