How to implement a conditional Upsert stored procedure?

Question

I'm trying to implement your basic UPSERT functionality, but with a twist: sometimes I don't want to actually update an existing row.

Essentially I'm trying to synchronize some data between different repositories, and an Upsert function seemed like the way to go. So based largely on Sam Saffron's answer to this question, as well as some other research and reading, I came up with this stored procedure:

(note: I'm using MS SQL Server 2005, so the MERGE statement isn't an option)

CREATE PROCEDURE [dbo].[usp_UpsertItem] 
    -- Add the parameters for the stored procedure here
    @pContentID varchar(30) = null, 
    @pTitle varchar(255) = null,
    @pTeaser varchar(255) = null 
AS
BEGIN
    -- SET NOCOUNT ON added to prevent extra result sets from
    -- interfering with SELECT statements.
    SET NOCOUNT ON;

    BEGIN TRANSACTION

     UPDATE dbo.Item WITH (SERIALIZABLE)
     SET Title = @pTitle,
      Teaser = @pTeaser
     WHERE ContentID = @pContentID

     IF @@rowcount = 0
      INSERT INTO dbo.Item (ContentID, Title, Teaser)
      VALUES (@pContentID, @pTitle, @pTeaser)

    COMMIT TRANSACTION
END

I'm comfortable with this for a basic Upsert, but I'd like to make the actual update conditional on the value of another column. Think of it as "locking" a row so that no further updates may be made by the Upsert procedure. I could change the UPDATE statement like so:

UPDATE dbo.Item WITH (SERIALIZABLE)
SET Title = @pTitle,
    Teaser = @pTeaser
WHERE ContentID = @pContentID
AND RowLocked = false

But then the subsequent Insert would fail with a unique constraint violation (for the ContentID field) when it tries to insert a row that already exists but wasn't updated because it was "locked".

So does this mean that I no longer have a classic Upsert, i.e. that I'll have to select the row every time to determine whether it can be updated or inserted? I'm betting that's the case, so I guess what I'm really asking for is help getting the transaction isolation level correct so that the procedure will execute safely.

Answer 1

You could switch the order of the update/insert around. So you do the insert within a try/catch and if you get a constraint violation then do the update. It feels a little dirty though.

Answer 2

I slapped together the following script to proof this trick I used in years past. If you use it, you'll need to modify it to suit your purposes. Comments follow:

/*
CREATE TABLE Item
 (
   Title      varchar(255)  not null
  ,Teaser     varchar(255)  not null
  ,ContentId  varchar(30)  not null
  ,RowLocked  bit  not null
)


UPDATE item
 set RowLocked = 1
 where ContentId = 'Test01'

*/


DECLARE
  @Check varchar(30)
 ,@pContentID varchar(30)
 ,@pTitle varchar(255)
 ,@pTeaser varchar(255)

set @pContentID = 'Test01'
set @pTitle     = 'TestingTitle'
set @pTeaser    = 'TestingTeasier'

set @check = null

UPDATE dbo.Item
 set
   @Check = ContentId
  ,Title  = @pTitle
  ,Teaser = @pTeaser
 where ContentID = @pContentID
  and RowLocked = 0

print isnull(@check, '<check is null>')

IF @Check is null
    INSERT dbo.Item (ContentID, Title, Teaser, RowLocked)
     values (@pContentID, @pTitle, @pTeaser, 0)

select * from Item

The trick here is that you can set values in local variables within an Update statement. Above, the "flag" value gets set only if the update works (that is, the update criteria are met); otherwise, it won't get changed (here, left at null), you can check for that, and process accordingly.

As for the transaction and making it serializable, I'd like to know more about what must be encapsulated within the transaction before suggesting how to proceed.

-- Addenda, follow-up from second comment below -----------

Mr. Saffron's ideas are a thorough and solid way of implementing this routine since your primary keys are defined outside and passed into the database (i.e. you're not using identity columns--fine by me, they are often overused).

I did some more testing (added a primary key constraint on column ContentId, wrap the UPDATE and INSERT in a transaction, add the serializable hint to the update) and yes, that should do everything you want it to. The failed update slaps a range lock on that part of the index, and that will block any simultaneous attempts to insert that new value in the column. Of course, if N requests are submitted simultaneously, the "first" will create the row, and it will be immediately updated by the second, third, etc.--unless you set the "lock" somewhere along the line. Good trick!

(Note that without the index on the key column, you'd lock the entire table. Also, the range lock may lock the rows on "either side" of the new value--or maybe they won't, I didn't test that one out. Shouldn't matter, since the duration of the operation should [?] be in single-digit milliseconds.)

Answer 3

A very common problem. Some approaches do not hold up under high concurrency. Described and stress tested here:

Stress testing UPSERTs

Defensive database programming: eliminating IF statements.

In such cases it is not enough to just write some code, you need to expose it to high concurrency. For example, I am not sure that I understood what CptSkippy recommends, but the following demonstrates how to stress test. Set up a table and a procedure:

CREATE TABLE [dbo].[TwoINTs](
      [ID] [int] NOT NULL,
      [i1] [int] NOT NULL,
      [i2] [int] NOT NULL,
      [i3] [int] NOT NULL
);
CREATE PROCEDURE dbo.SaveTwoINTs(@ID INT, @i1 INT, @i2 INT)
AS
BEGIN
      SET NOCOUNT ON;
      SET XACT_ABORT OFF;
      SET TRANSACTION ISOLATION LEVEL READ COMMITTED;
      DECLARE @ret INT;
      SET @ret=0;
      BEGIN TRAN; 
IF EXISTS(SELECT 1 FROM dbo.TwoINTs WHERE ID=@ID) BEGIN
      UPDATE dbo.TwoINTs WITH (SERIALIZABLE)
      SET i1=i1+@i1, i2=i2+@i2 WHERE ID=@ID;
      SET @ret=@@ERROR;
END ELSE BEGIN
     INSERT INTO dbo.TwoINTs(ID, i1, i2, i3)VALUES(@ID, @i1, @i2, @i1);
      SET @ret=@@ERROR;
END;
COMMIT;
RETURN @ret;
END
GO

Set up two loops that execute that procedure:

CREATE PROCEDURE Testers.UpsertLoop1
AS
BEGIN
DECLARE @ID INT, @i1 INT, @i2 INT, @count INT, @ret INT;
SET @count = 0;
WHILE @count<50000 BEGIN
      SELECT @ID = COALESCE(MAX(ID),0) + 1 FROM dbo.TwoInts;
    EXEC @ret=dbo.SaveTwoINTs @ID, 1, 0;
      SET @count = @count + 1;
END;
END;
GO
CREATE PROCEDURE Testers.UpsertLoop2
AS
BEGIN
DECLARE @ID INT, @i1 INT, @i2 INT, @count INT, @ret INT;
SET @count = 0;
WHILE @count<50000 BEGIN
      SELECT @ID = COALESCE(MAX(ID),0) + 1 FROM dbo.TwoInts;
    EXEC @ret=dbo.SaveTwoINTs @ID, 0, 1;
      SET @count = @count + 1;
END;
END;

Execute these procedures in two tabs and see for yourself that you get a lot of errors:

Testers.UpsertLoop1 --run in one tab
Testers.UpsertLoop1 --run in one tab

Msg 2601, Level 14, State 1, Procedure SaveTwoINTs, Line 15
Cannot insert duplicate key row in object 'dbo.TwoINTs' with unique index 'UNQ_TwoInts_ID'.
The statement has been terminated.

Follow the links that I provided to see the approaches that actually work under concurrency.

Answer 4

CREATE PROCEDURE [dbo].[usp_UpsertItem] -- Add the parameters for the stored procedure here @pContentID varchar(30) = null, @pTitle varchar(255) = null, @pTeaser varchar(255) = null AS BEGIN -- SET NOCOUNT ON added to prevent extra result sets from -- interfering with SELECT statements. SET NOCOUNT ON;

BEGIN TRANSACTION
    IF EXISTS (SELECT 1 FROM dbo.Item WHERE ContentID = @pContentID
             AND RowLocked = false)
       UPDATE dbo.Item 
       SET Title = @pTitle, Teaser = @pTeaser
       WHERE ContentID = @pContentID
             AND RowLocked = false
    ELSE IF NOT EXISTS (SELECT 1 FROM dbo.Item WHERE ContentID = @pContentID)
            INSERT INTO dbo.Item (ContentID, Title, Teaser)
            VALUES (@pContentID, @pTitle, @pTeaser)

COMMIT TRANSACTION

END

Answer 5

BEGIN TRANSACTION

IF EXISTS(SELECT 1 FROM dbo.Item WHERE ContentID = @pContentID)
     UPDATE dbo.Item WITH (SERIALIZABLE)
     SET Title = @pTitle, Teaser = @pTeaser
     WHERE ContentID = @pContentID
     AND RowLocked = false
ELSE
     INSERT INTO dbo.Item
          (ContentID, Title, Teaser)
     VALUES
          (@pContentID, @pTitle, @pTeaser)

COMMIT TRANSACTION