data-loading

Maintain valid records in a table

I have a table which holds flight schedule data. Every schedule have an effective_from & effective_to date. I load this table from flat file which don't provide me an effective_from and effective_to date. So at the time of loading I ask this information from user. Suppose user gave from date as current date and to date as 31st March. No...

Complex processing in Stored procedures Vs .net application

We are building a new application in .net 3.5 with SQL server database. The database is fairly large having around 60 tables with loads on data. The .net application have functionality to bring data into this database from data entry and from third party systems. After all the data is available in database the system have to do lots of ...

Question about loading data in a Rails migration

I created number of migrations starting with a definition for a Posts table. class CreatePosts < ActiveRecord::Migration def self.up create_table :posts do |t| t.column "title", :string, :limit => 100, :default => "", :null => false t.column "content", :text, :null => false t.column "author", :string, :limit =>...

Asynchronous data loading in Entity-Framework?

Did anyone hear about asynchronous executing of an EF query? I want my items control to be filled right when the form loads and the user should be able to view the list while the rest of the items are still being loaded. Maybe by auto-splitting the execution in bulks of items (i.e. few queries for each execution) all in same connection...

What's the fastest way to save data and read it next time in a IPhone App ?

I have the following problem: In my dictionary IPhone app I need to save an array of strings which actually contains about 125.000 distinct words; this transforms in aprox. 3.2Mb of data. The first time I run the app I get this data from an SQLite db. As it takes ages for this query to run, I need to save the data somehow, to read it fa...

Does Pentaho Kettle have a way to accept JMS messages?

Does Pentaho's ETL system, Kettle (http://kettle.pentaho.org/) have a plugin to accept information from JMS messages? I'd like to set up a job that can read messages each containing a hash, extract certain data, and stuff it into a table. Has anyone had any experience with this? Is this a good idea? Any pitfalls? Thanks. ...

Good place for data loading scripts that rely on rails?

In your experience, where is the best place to place scripts that run data loading jobs, but which rely on rails? In my project they are in the model folder, but that adds a lot of code to the model folder and won't rails load it all into memory when the server is run (unnecessarily)? The lib/ folder looks good, but those don't have rail...

Data load to huge partitioned table

I have a huge table. First range partitioned by price_date, then hash partitioned by fund_id. The table has 430 million rows. Every day I have a batch job in which insert 1.5 million to 3 million rows, every day. We are looking the for enable and disable local indexes(not all indexes but based on data which partitions are touched by d...

Using SQL Server spatial types in SSIS data load

Hi there, I am trying to load a file using the SQL Server Import Data function - appending rows to an existing table. my existing table is created like this: CREATE TABLE [dbo].[load]( [Long] [varchar](50) NULL, [Lat] [varchar](50) NULL, [Geog] AS ([geography]::STGeomFromText(((('POINT('+[Long])+' ')+[Lat])+')',(4326))) )...