How to model an object with references to arbitrary number of arbitrary field types? (django orm)

Question

I'd like to define a set of model/objects which allow for one to represent the relationship: field_set has many fields where fields are django.db.model field objects (IPAddressField, FilePathField etc).

My goals is to have a ORM model which supports the following type of 'api'.

From a controller view lets say:

# Desired api below
def homepage(request):
  from mymodels.models import ProfileGroup, FieldSet, Field

  group = ProfileGroup()
  group.name = 'Profile Information'
  group.save()

  geographicFieldSet = FieldSet()
  # Bind this 'field set' to the 'profile group'
  geographicFieldSet.profilegroup = group
  address_field = Field()
  address_field.name = 'Street Address'
  address_field.f = models.CharField(max_length=100)
  # Bind this field to the geo field set
  address_field.fieldset = geographicFieldSet 

  town_field = Field()
  town_field.name = 'Town / City'
  town_field.f = models.CharField(max_length=100)
  # Bind this field to the geo field set
  town_field.fieldset = geographicFieldSet 

  demographicFieldSet = FieldSet()
  demographicFieldSet.profilegroup = group
  age_field = Field()
  age_field.name = 'Age'
  age_field.f = models.IntegerField()
  # Bind this field to the demo field set
  age_field.fieldset = demographicFieldSet 

  # Define a 'weight_field' here similar to 'age' above.

  for obj in [geographicFieldSet, town_field, address_field, 
              demographicFieldSet, age_field, weight_field]:
    obj.save()


  # Id also add some methods to these model objects so that they 
  # know how to render themselves on the page...

  return render_to_response('page.templ', {'profile_group':group})

Essentially I want to support 'logically grouped fields' since I see myself supporting many 'field sets' of different types thus my desire for a meaningful abstraction.

Id like to define this model so that I can define a group of fields where the # of fields is arbitrary as is the field type. So I may have a field group 'Geographic' which includes the fields 'State' (CharField w/ choices), 'Town' (TextField) etc.

Heres what Ive come up with so far:

class ProfileGroup(models.Model):
  name = models.CharField(max_length=200)

# FieldSets have many Fields
class FieldSet(models.Model):
  name = models.CharField(max_length=200)
  profilegroup = models.ForeignKey(ProfileGroup)

class Field(models.Model):
  f = models.Field()
  fieldset = models.ForeignKey(FieldSet)

Though using these models produces an error in the shell and ultimately doesnt allow me to store arbitrary fields.

In [1]: from splink.profile_accumulator.models import Field, FieldSet, ProfileGroup
In [2]: import django.db
In [3]: profile_group = ProfileGroup()
In [4]: profile_group.name = 'profile group name'
In [5]: profile_group.save()
In [6]: field_set = FieldSet()
In [7]: field_set.name = 'field set name'
In [8]: field_set.profilegroup = profile_group
In [9]: field_set.save()
In [10]: field = Field()
In [11]: field.name = 'field name'
In [12]: field.f = django.db.models.FileField()
In [13]: field.save()
---------------------------------------------------------------------------
ProgrammingError                          Traceback (most recent call last)

/var/www/splinkpage.com/splinkpage.pinax/splink/<ipython console> in <module>()

/usr/lib/pymodules/python2.5/django/db/models/base.pyc in save(self, force_insert, force_update)
    309             raise ValueError("Cannot force both insert and updating in "
    310                     "model saving.")
--> 311         self.save_base(force_insert=force_insert, force_update=force_update)
    312
    313     save.alters_data = True

/usr/lib/pymodules/python2.5/django/db/models/base.pyc in save_base(self, raw, cls, force_insert, force_update)
    381             if values:
    382                 # Create a new record.
--> 383                 result = manager._insert(values, return_id=update_pk)
    384             else:
    385                 # Create a new record with defaults for everything.

/usr/lib/pymodules/python2.5/django/db/models/manager.pyc in _insert(self, values, **kwargs)
    136 
    137     def _insert(self, values, **kwargs):
--> 138         return insert_query(self.model, values, **kwargs)
    139 
    140     def _update(self, values, **kwargs):

/usr/lib/pymodules/python2.5/django/db/models/query.pyc in insert_query(model, values, return_id, raw_values)
    890     part of the public API.
    891     """
    892     query = sql.InsertQuery(model, connection)
    893     query.insert_values(values, raw_values)
--> 894     return query.execute_sql(return_id)

/usr/lib/pymodules/python2.5/django/db/models/sql/subqueries.pyc in execute_sql(self, return_id)
    307
    308     def execute_sql(self, return_id=False):
--> 309         cursor = super(InsertQuery, self).execute_sql(None)
    310         if return_id:
    311             return self.connection.ops.last_insert_id(cursor,

/usr/lib/pymodules/python2.5/django/db/models/sql/query.pyc in execute_sql(self, result_type)
   1732
   1733         cursor = self.connection.cursor()
-> 1734         cursor.execute(sql, params)
   1735
   1736         if not result_type:

/usr/lib/pymodules/python2.5/django/db/backends/util.pyc in execute(self, sql, params)
     17         start = time()
     18         try:
---> 19             return self.cursor.execute(sql, params)
     20         finally:
     21             stop = time()

ProgrammingError: can't adapt

So Im wondering if this is totally the wrong approach or if I need to use django's model classes a bit differently to get what I want.

Answer 1

In SQL there is no such thing as a table with variable number of columns or variable type columns. Also, Django does not modify database layout at run time - i.e. does not call ALTER TABLE statements - as far as I know.

In django data models must be completely defined before you run your application.

You might find this doc page relevant for use of "many-to-one" relationships.

Example:

#profile
class ProfileGroup(models.Model):
    ...

#fieldset
class FieldSet(models.Model):
    profile = models.ForeignKey(ProfileGroup)

#field 1
class Address(models.Model)
    owner = models.ForeignKey(FieldSet,related_name='address')
    #related name above adds accessor function address_set to Profile
    #more fields like street address, zip code, etc

#field 2
class Interest(models.Model)
    owner = models.ForeignKey(FieldSet,related_name='interest')
    description = models.CharField()

#etc.

Populate and access fields:

f = FieldSet()
f.interest_set.create(description='ping pong')
f.address_set.create(street='... ', zip='... ')
f.save()

addresses = f.address_set.all()
interests = f.interest_set.all()
#there are other methods to work with sets

sets in this case emulate variable fields in a table Profile. In the database, however, interest and address data is stored in separate tables, with foreign key links to Profile.

If you want to access all that with one accessor function - you could write something that wraps calls to all related sets.

Even though you can't modify model on the fly, you can add new models then issue

manage.py syncdb

This will create new tables in the db. However you won't be able to modify fields in existing tables with 'syncdb' - django doesn't do it. You'll have to enter SQL commands manually for that. (supposedly web2py platform handles this automatically, but unfortunately web2py is not well documented yet, but it might be a cut above django in terms of quality of API and is worth taking a look at)

Answer 2

I see several problems with the code. First, with this class definition:

class Field(models.Model):
  f = models.Field()
  fieldset = models.ForeignKey(FieldSet)

The class models.Field is not supposed to be used directly for a field definition. It is a base class for all field types in Django so it lack specifics for a particular field type to be useful.

The second problem is with the following line:

In [12]: field.f = django.db.models.FileField()

When you assign to attribute f of your Field instance, you are supposed to give a specific value to be saved to the database. For example, if you used CharField for Field.f definition, you would assign a string here. models.Field has no specific assignable values though. You are trying to assign something that is clearly not possible to save to the DB. It is modles.FileField definition.

So, Django has a hard time to "adjust" the value you are assigning to the field attribute for two reasons. First, the are no values defined for models.Field type to assign as it is an "abstract class", or a base class for specific field type definitions. Second, you can not assign "field definition" to an attribute and hope that it is going to be saved to a DB.

I understand your confusion. You are basically trying to do impossible thing both from the DB and Django's points of view.

I suppose there could be a solution for your design problem though. If you describe what you are trying to achieve in detail, somebody could probably give you a hint.