For a big commercial online shop my company needs to start a business intelligence project. We have done the online shop; there are probably only a couple of data sources, so the integration i think it's quite easy. We're creating the data warehouse database, revisiting the actual database from another point of view. I start to look...
Hi, anybody have used Jitterbit as an ETL tool? any reviews/tutorials? ...
Is there a direct route that is pretty straight forward? (i.e. can SQL Server read XML) Or, is it best to parse the XML and just transfer it in the usual way via ADO.Net either as individual rows or perhaps a batch update? I realize there may be solutions that involve large complex stored procs--while I'm not entirely opposed to this,...
I have three denormalized tables that I have to take at face value (data comes from some external resource). The three tables have different definitions, but they each describe the same object from different perspectives. object1 A B object2 A object3 B C object4 C The only commonality between these tables i...
I have a csv file with 350,000 rows, each row has about 150 columns. What would be the best way to insert these rows into SQL Server using ADO.Net? The way I've usually done it is to create the SQL statement manually. I was wondering if there is any way I can code it to simply insert the entire datatable into SQL Server? Or some shor...
I need to import data from an Excel spreadsheet into SQL Server, but the data isn't in a relational/normalized format so the import wizard isn't going to cut it (as far as I know). The data is in this format: Category SubCategory Name Description Category#1 SubCategory#1 Product#1 Description#1 Category#1 SubCatego...
I work for a large Canadian retailer, and we're currently considering using CozyRoc for a large ETL project of ours. I was wondering if anyone could recommend the software, ideally through an e-mail chain or over the phone. ...
I have 2 tables (~ 4 million rows) that I have to do insert/update actions on matching and unmatching records. I am pretty confused about the method I have to use for incremental load. Should I use Lookup component or new sql server merge statement? and will there be too much performance differences? ...
I've just started at a new company in outsourced communications (e.g. print and mail, email, fax). One of the requirements is to process clients data and get it ready for mailing. For recurring jobs, this is easy using an ETL tool linked in with some addressing software, but for adhoc stuff it's a bit overkill. I've used inhouse develop...
I am wondering how SSIS deals with transactions with large data sets. I have a 'large' data set of about 150,000 rows, each of which needs to be validated against business rules as part of an ETL from a staging database to the live database. If any of the records fail their business rules, no records should end up in the live database (...
I have a few SSIS packages that were password-protected (their protection level is apparently EncryptAllWithPassword) by a developer who left the company and can't be reached anymore, and trying to open them gives the following error since the password can't be supplied: Error loading 'Package.dtsx' : Failed to remove package protect...
Hi Everyone, I'm looking for some decent examples/samples using SSIS to do some ETL from one SQL Server database to another not necessarily within the same instance. The idea is to migrate rows of data with their heirarchies (relationships) from one OLTP database to another. There are some advantages SSIS offers us which makes it a go...
I am looking for a scripting language that meets the following criteria: Well Documented Implemented in Java Understands Datatypes (can handle longs, strings, etc) Is fully extensible Any suggestions? Currently, I'm using Javascript, but the Doubles that it sets numbers to is not big enough for the data work I'm doing. Thanks! ...
I'm looking for the best solution to allow our users to upload XLS spreadsheet so that they can be used to populate tables in our data warehouse (DW). Our users are heavy Business Object (BO) users, and BO lets you export to XLS. When they have data in a spreadsheet that needs to be loaded to the DW, they need a process to upload the d...
I don't know the first thing about Informatica but I am looking for ways to resolve duplicating business logic that for inserting and updating records in a table. The problem is doing it in an efficient manner. 1) We have web pages that insert, update and delete records one at a time. 2) We have Informatica ETL load programs that take ...
I am trying to get the new System.Diagnostics.Eventing event logging working in a simple .Net app before integrating it into my application. Working off of this page, I created a manifest, built a simple app that fires an event and registered the provider to see it in the event viewer. I don't get any errors in event viewer or my sampl...
I've got a System.Diagnostics.Eventing.EventProvider class trying to write to a custom event log. The event log was created using a manifest file and the procedure layed out on this page. You can see the new provider and channel in the windows Event Viewer, but for some reason no events that get written seem to show up there. When I ch...
When loading tables from SAS to Teradata, SAS loads the data (usually using the FASTLOAD facility) and then continues down the script. However, I often get critical errors because SAS says the data is loaded, but Teradata is still assembling the data within the table. So the data is in the database, but not ready to be used. I have yet...
I have implemented delta detection while loading data warehouse from transaction systems using an identity column or date-time column in source transaction tables. When data needs to be extracted next time, the maximum date-time value extracted last time is used in the filter of extraction query to identify new or changed records. This w...
i have a page that allow user to upload an excel file and insert the data in excel file to the SQL Server. Now i have a small issue that, there is a column in excel file with values, such as "001", "029", "236". When it's insert to the SQL Server, the zero in front will be ignored in SQL, so the data would become "1", "29", "239". The da...