SQL Alchemy default value function for simulating autoincrement within a unique group of parent-child records

Question

I have a small problem that I think should be easily handled by SQL Alchemy but I can't seem to get it right. I have two tables with one being a parent table and the other a child table. For each child record it needs a unique ID but only with the context of the unique parent record.

I am using the Declarative Base approach.

I setup the parent child relationship using the FKs and the relation function. What I'm trying to achieve is to get something like a pseudo autoincrement function that will lookup the max CategoryID value within the Category unique name group and increment it up one. I've tried using various default functions but the problem I run into is the inability to specify the CategoryUniqueName at the time of insertion. I can't find a way to pass the current value of CategoryItems.CategoryUniqueName so that the lookup query has the correct filter applied when trying to select something like func.max(CategoryItems.CategoryID). If I hardcode a query it works just fine. This is what I'm thinking should work but, again, I can't find a way to specify the unique value for the filter.

unique_group='my_group'
result=con.query(func.max(CategoryItems.CategoryID)).filter(and_(
        CategoryItems.CategoryUniqueName==unique_group, 
    )).one()

The classes are shown below. Much appreciate some guidance on how to accomplish this inside standard SQL Alchemy. I know I could always lookup the value and simply specify it directly within the same transaction but I'm trying to come up with a stand-alone SQL Alchemy approach that does not require additional logic elsewhere.

class Category(Base):
    __tablename__ = 'parent_table'
    __table_args__ = {'mysql_engine':'InnoDB', 'useexisting':True}

    CategoryUniqueName = Column(Unicode(255), primary_key=True)
    CategoryGroupName = Column(Unicode(255), nullable=False)
    CategoryGroupMemo = Column(UnicodeText)
    SortOrder = Column(Integer, index=True)
    IsLocked = Column(Boolean, default=0)

class CategoryItems(Base):
    __tablename__ = 'child_table'
    __table_args__ = {'mysql_engine':'InnoDB', 'useexisting':True}

    CategoryUniqueName = Column(Unicode(255), ForeignKey(Category.CategoryUniqueName), primary_key=True)
    CategoryID = Column(Integer, primary_key=True, autoincrement=False)
    CategoryName = Column(Unicode(255), nullable=False, index=True)
    CategoryMemo = Column(UnicodeText)
    CategoryImage = Column(Unicode(255))
    CategoryFlex1 = Column(Unicode(255), index=True)
    CategoryFlex2 = Column(Unicode(255), index=True)
    CategoryFlex3 = Column(Unicode(255), index=True)
    SortOrder = Column(Integer, index=True)

    category_group = relation(
        Category, 
        backref=backref(
            'items', 
            order_by=SortOrder, 
            collection_class=ordering_list('SortOrder'), 
            cascade="all, delete, delete-orphan"
    ))

Answer 1

So, what you're trying to achieve is have each set of CategoryItems with a different CategoryUniqueName have their CategoryId auto-incremented separately?

If that's right, your current approach (get the current max of CategoryId in the subset of CategoryItems you're adding to) is broken. It has an intrinsic race condition: concurrent insertions will use the same CategoryId.

Do you really need your CategoryId incremented separately? Why not just use the normal auto-increment feature. The sequence CategoryId's for a given CategoryUniqueName will have holes, but is that really a problem?

If you require successive sequence numbers, you will need to prevent race condition using some manual locking.

Answer 2

I see 3 ways to go:

1. The most obvious and well documented. Create a mapper extension with before_insert() hook replacing inserted parameter.
2. Pass function as default argument. This function is called with context parameter with all data you need: context.compiled_parameters[0]['CategoryUniqueName'], context.connection.
3. Pass FetchedValue() in server_default parameter and use trigger to do the job server side.

All these solutions have race condition mentioned by ddaa. In case of race condition your code won't break database state, but will fail with exception when primary key is defined properly (it's not true for your code!). It might be acceptable for some application to fail (show 500 page in web application) in some rare cases.

Note, that you have defined a CategoryID as primary key. This won't allow reuse of the same number for different values of CategoryUniqueName column. You have to change it to composite primary index for 2 columns.

Answer 3

Thanks for the insight Denis, you are spot on. I played with both options 1 and 2 and they work very nicely. The context parameter was the key for option 2. I didn't realize it was being passed automatically. One thing I did note was option 1 would introduce a race condition even in the context of a single user submit of multiple records. I think this has something to do with the flush and save timing. However option 2 works brilliantly.

This is the little function that gets called from the default parameter now:

def getNextId(context):
    unique_name=context.compiled_parameters[0]['CategoryUniqueName']
    sql = """
        SELECT MAX(CategoryID)
        FROM child_table
        WHERE CategoryUniqueName='%s'""" % (unique_name, )

    result = context.connection.execute(sql).fetchone()[0]
    if result > 0:
         return result + 1
    else:
        return 1