Performing Insert OR Update (upsert) on sql server compact edition

Question

I have c# project that is using sqlserver compact edition and entity framework for data access. I have the need to insert or update a large amount of rows, 5000+ or more to the db, so if the key exists update the record if not insert it. I can not find a way to do this with compact edition and EF with out horrible performance, ie taking 2 mins plus on a core i7 computer. I have tried searching for the record to see if it exists then inserting if not or update if it does, the search is the killer on that. I have tried compiling the search query and that only gave a small improvement. Another thing ive tried is inserting the record in a try catch and if it fails update, but that forces me to savechanges on every record to get the exception as opposed to at the end which is a performance killer. Obviously I can't use stored procedures since it is compact edition. Also I've looked at just executing t-sql directly somehow on the db, but lack of process statements in compact seems to rule that out. I've searched the world over and out of ideas. I really wanted to use compact if I can over express for the deployment benefits and ability to prevent the user from digging around the db. Any suggesitons would be appreciated.

Thanks

Answer 1

Given your problem statement, I'm going to guess that this software assumes a relatively beefy environment. Have you considered taking the task of determining off of sqlce and doing it on your own? Essentially, grab a sorted list of all the IDs(keys?) from the relevant table and checking every object key against that list before queueing it for insertion?

This makes a few assumptions that would be bad news with a typical DB, but that you can probably get away with in sqlce. E.g., it assumes that rows won't be inserted or significantly modified by a different user while you're performing this insert.

If the list of keys is too long to reasonably hold in memory for such a check, I'm afraid I'd say that sqlce just might not be the right tool for the job. :(

Answer 2

Maybe you could obtain the result you seek by using simple queries. Let's say the the table you want to insert into or update is like this

TABLE original
     id integer,
     value char(100)

first you could create a temporary table with the new values (you can use a SELECT INTO or other ways to create it)

TABLE temp
    id integer,
    value char(100)

now, you need to do two things, update the rows in original and then insert the new values

UPDATE original 
SET original.value = temp.value
FROM original, temp
WHERE original.id = temp.id

INSERT INTO original 
SELECT * from temp 
WHERE temp.id not IN (select o.id from original o)

Answer 3

I'm not sure if this is feasible or not, as I haven't used the Entity Framework, but have you tried running the update first and checking the rowcount -- inserting if no rows were updated? This may be faster than catching exceptions. It's generally a bad practise to use exceptions for control flow, and often slows things down dramatically.

If you can write the SQL directly, then the fastest way to do it would be to get all the data into a temporary table and then update what exists and insert the rests (as in Andrea Bertani's example above). You should get slightly better results by using a left join on the original table in the select in your insert, and excluding any rows with values from the original table that are not null:

INSERT INTO original
SELECT * FROM temp
LEFT JOIN original ON original.id = temp.id
WHERE original.id IS NULL

Answer 4

I would recommend using SqlCeResultSet directly. You lose the nice type-safeness of EF, but performance is incredibly fast. We switched from ADO.NET 2.0-style TypeDataSets to SqlCeResultSet and SqlCeDataReader and saw 20 to 50 times increases in speed.

Answer 5

SQL Server compact edition is pretty early in development at this point. Also, depending on your device, memory-disk access can be pretty slow, and SQLCE plus .NET type-safety overhead is pretty intensive. It works best with a pretty static data store.

I suggest you either use a lighter-weight API or consider SQLite.

Answer 6

See SqlCeResultSet. For a .NETCF project I removed almost all sql code in favor of this class. Just search for "SqlCeResultSet" here and msdn.

A quick overview:

1. Open the resultSet. If you need seek (for existence check) you will have to provide an index for the result set.

2. Seek on the result set & read to check whether you found the row. This is extremely fast even on tables with tens of thousands rows (because the seek uses the index).

3. Insert or update the record (see SqlCeResultSet.NewRecord).

---

We have successfully developed a project with a sqlce database with a main product table with over 65000 rows (read/write with 4 indexes).

Answer 7

When we're using SQL CE (and SQL 2005 Express for that matter) we always call an update first and then call an insert if the udate gives a row count of 0. This is very simple to implement and does not require expensice try..catch blocks for control flow.