bulk

How to change slow parametrized inserts into fast bulk copy (even from memory)

I had someting like this in my code (.Net 2.0, MS SQL) SqlConnection connection = new SqlConnection(@"Data Source=localhost;Initial Catalog=DataBase;Integrated Security=True"); connection.Open(); SqlCommand cmdInsert = connection.CreateCommand(); SqlTransaction sqlTran = connection.BeginTransaction(); cmdInsert.Transaction = sq...

Avoid being blocked by web mail companies for mass/bulk emailing ?

Our company is sending out a lot of emails per day and planning to send even more in future. (thousands) Also there are mass mailouts as well in the ten thousands every now and then. Anybody has experience with hotmail, yahoo (web.de, gmx.net) and similar webmail companies blocking your emails because "too many from the same source in a...

Is there a simple way to do bulk file text substitution in place?

First of all, I'm a Perl newbie, so please be gentle =) I've been trying to code a Perl script to substitute some text on all source files of my project. I'm in need of something like: perl -p -i.bak -e "s/thisgoesout/thisgoesin/gi" *.{cs,aspx,ascx} But that parses all the files of a directory recursively. I just started a script: ...

Bulk Translation Of Table Contents

I'm currently performing a migration operation from a legacy database. I need to perform migration of millions of originating rows, breaking the original content apart into multiple destination parent / child rows. As it's not a simple 1 to 1 migration and the the resulting rows are parent / children row based on identity generated key...

How to add more than one test object from testobject map to the script.

Test which I have to automate: In an applicationmy I have to create 1000 objects. Then I have to go to each object and I have to put some value. Using script I created one object and then going to this object and putting value. Now in object creation step I put a for loop which created 1000 objects of different name. Now using "Add Ob...

Building a bulk mail sender

Hi, I want to build an application that will allow my customers to send marketing information by e-mail. This will be a carefully monitored tool used for legitimate bulk mailing only. It's going to have all of the necessary 'unsubscribe' functionality etc. The solution will be built using VB.NET. My question relates to the best way to...

What is the fastest way to parse text with custom delimiters and some very, very large field values in C#?

I've been trying to deal with some delimited text files that have non standard delimiters (not comma/quote or tab delimited). The delimiters are random ASCII characters that don't show up often between the delimiters. After searching around, I've seem to have only found no solutions in .NET will suit my needs and the custom libraries t...

Faster bulk inserts in sqlite3?

I have a file of about 30000 lines of data I want to load into a sqlite3 database. Is there a faster way that generating insert statements for each line of data? The data is space delimited and maps directly to the sqlite3 table. Is there any sort of bulk insert method for adding volume data to a database? Has anyone devised some devio...

MS SQL Server - Bulk Insert Across a Network

I have an application that uses MS SQL Server for which I'll need to do a bulk insert from a file. The sticking point is that the database and my application will be hosted on separate servers. What is the best way to do a bulk insert across a network? Two ideas I'd come up with so far: From the app server, share a directory that th...

SQL Server Bulk Insert failing when called from .NET SqlCommand

I have a stored procedure which does bulk insert on a SQL server 2005 database. When I call this stored procedure from some SQL (passing in the name of a local format file and data file) it works fine. Every time. However, when this same stored procedure gets called from C# .NET 3.5 code using SqlCommand.ExecuteNonQuery it works intermi...

How to save data from a semicolon delimited file using Bulk Insert in SQL Server 2005 and BCP 9?

Here is my sample data: 1;a;b;c;; 2;d;e;f;; 3;g;h;i;; 4;j;k;l;; 5;m;n;o;; 6;p;q;r;; Here is my sample format file (BCP 9): 9.0 7 1 SQLCHAR 0 0 "" 0 x Latin1_General_CI_AS 2 SQLCHAR 0 0 ";" 2 i Latin1_General_CI_AS 3 SQLCHAR 0 0 ";" 3 s Latin1_General_CI_AS 4 SQLCHAR 0 0 ";" 4 t Latin1_General_CI_AS 5 SQLCHAR 0 0 ";" 5 u Latin1_Genera...

Is there a bulk email plugin for Rails apps?

Does anyone know of a plugin or something that can be used to send bulk emails for a Rails app? Specifically, I'd like to be able to pass an HTML email file to a rake task or something and have it emailed out to everyone who has signed up to my site and checked the "please send me info about XXX" box. I wrote kind of a hacked-together ...

Bulk upgrade VS2005 solutions to VS2008

Is it possible to bulk upgrade (many at the same time) VS 2005 projects to VS 2008. I know that I can open one at a time, however, I would like to select say 10 at a time to upgrade and add to a new solution. ...

XHTML bulk Validator for Windows?

I have several hundred (static - no includes or server-side parsing) HTML pages. I would like to validate the HTML (XHTML 1.0 Transitional), but the idea of plugging each of them individually into the W3C Validator - or even using a Firefox plug-in to open each one - fills me with dread. Does anyone know of any (Windows) apps that can d...

SQL Server recovery and bulk recovery mode confusions

Hello everyone, I have 2 confusions about SQL Server recovery and bulk recovery mode. whether recovery process itself will generate transaction log (here what I mean the logs are the logs generated by recovery process -- if any, and not generated by normal database DML/DDL operations)? Why? in the bulk recovery mode, whether the under...

How do I bulk upload to s3?

I recently refactored some of my code to stuff rows into a db using 'load data' and it works great -- however for each record I have I must upload 2 files to s3 -- this totally destroys the magnificent speed upgrade that I was obtaining. Whereas I was able to process 600+ of these documents/second they are now trickling in at 1/second be...

What is the fastest way to update many rows of data with different values in sql server

My company recently had a problem where we needed to update 64,000+ rows of data fairly regularly, where each row would be update with unique values. I have come up with a good answer, which I am posting here for others' reference, but also to find feedback if there are other solutions. Clarification : Each row has unique values that a...

how does one upload data in bulk to a google appengine datastore?

I have about 4000 records that i need to upload to the datastore. They are currently in CSV format. I'd appreciate if someone would point me to or explain how to upload data in bulk to GAE. Thank you very much. Help appreciated. ...

SQL Bulk Insert error 4863

Hey guys, I am trying to import some data to SQL Server 2008 by means of Bulk Insert, but I've been getting a ton of conversion errors: Msg 4864, Level 16, State 1, Line 1 Bulk load data conversion error (type mismatch or invalid character for the specified codepage) for row 5902, column 2 (Type). OK so first things first: a) the da...

Microsoft Sql Management Studio, SQL Server Agent, Build a Insert Bulk with dynamic parameters.

Here is the syntax to start with (classified data omitted) BULK INSERT [dbName].[dbo].[data_200801] FROM '\\servername\wwwroot\path\Files\data_200904.csv' WITH ( FIELDTERMINATOR = ';', ROWTERMINATOR = '\n' ); This works fine as one of the step of the job. Where I hit a wall is that I need the file name...