SSIS - How do I use a resultset as input in a SQL task and get data types right?

Question

I am trying to merge records from an Oracle database table to my local SQL table.

I have a variable for the package that is an Object, called OWell.

I have a data flow task that gets the Oracle data as a SQL statment (select well_id, well_name from OWell order by Well_ID), and then a conversion task to convert well_id from a DT_STR of length 15 to a DT_WSTR; and convert well_name from a DT_STR of length 15 to DT_WSTR of length 50. That is then stored in the recordset OWell.

The reason for the conversions is the table that I want to add records to has an identity field: SSIS shows well_id as a DT_WSTR of length 15, well_name a DT_WSTR of length 50.

I then have a SQL task that connects to the local database and attempts to add records that are not there yet. I've tried various things: using the OWell as a result set and referring to it in my SQL statement. Currently, I have the ResultSet set to None, and the following SQL statment:

Insert into WELL (WELL_ID, WELL_NAME)
Select OWELL_ID, OWELL_NAME
from OWell
where OWELL_ID not in 
   (select WELL.WELL_ID from WELL)

For Parameter Mapping, I have Paramater 0, called OWell_ID, from my variable User::OWell. Parameter 1, called OWell_Name is from the same variable. Both are set to VARCHAR, although I've also tried NVARCHAR. I do not have a Result set.

I am getting the following error: Error: 0xC002F210 at Insert records to FLEDG, Execute SQL Task: Executing the query "Insert into WELL (WELL_ID, WELL_NAME) Select OWELL..." failed with the following error: "An error occurred while extracting the result into a variable of type (DBTYPE_STR)". Possible failure reasons: Problems with the query, "ResultSet" property not set correctly, parameters not set correctly, or connection not established correctly.

I don't think it's a data type issue, but rather that I somehow am not using the resultset properly. How, exactly, am I supposed to refer to that recordset in my SQL task, so that I can use the two recordset fields and add records that are missing?

Answer 1

To infer missing rows we either used a lookup task and then directed the unfound rows to an ordinary OLEDB destination (you just don't supply the identity column, obviously) or (where we were comparing a whole table) the SQLBI.com TableDifference component and routed the new rows to a similar OLEDB destination.

Individual INSERTs in SQL Command task aren't terribly quick.

Answer 2

Your problem is that you are trying to read an object variable into a sql task, and refer to that variable in the sql task.

To do what you are trying to do, you can use a foreach loop task. You can set the enumerator of a for each to an object (recordset) variable and map its columns to variables that you can then pass as parameters into your sql task. Your sql code in the example above has another flaw in that you are trying to reference a variable in your package as if it were a table in your database. You need to change your sql to be something like Insert into well(?,?)

This approach however leaves out the step where you can check to see if the records exists before you insert it. A better overall approach would be to do this all in a dataflow.

Do everything you are doing in your select from Oracle dataflow. At the last step, instead of using a recordset destination pointing to variable USER::OWell, add a lookup from the local sql table. Set your sql statement there to be select WELL.WELL_ID from WELL. On the columns tab in your lookup match Well_ID from your dataflow (fields on the left) to Well_ID from your lookup (fields on the right) by dragging the well_id field from the left to the right to form a connector between the boxes. At the bottom of the dialog box, click on Configure Error Output and set the error column value for the lookup output row to be Redirect Row. Choose OK to save and close this lookup. Next, add a oledb destination to the data flow and connect it to the error output of the lookup (the red arrow). Point the destination to the sql table and map the columns from the dataflow to the appropriate columns in the output table. This will pass the rows from the oracle dataflow that do not exist in the sql table into the bulk insert of the sql table.