batch-processing

Batch code to re-name set of template files by changing part of the name of one file (all files to copy changes to the same text).

I have created a filing system containing folders as templates for re-creating a set of information to be used when we accept bookings and need a batch command to rename all files within a directory but in a very specific way. So to be more precise the directory contains: one file is an invoice template (PDF form) and another is a job d...

quick linux question

I'm trying to do the following: $OOOPYTHON DocumentConverter.py .odt .pdf for all of the ODT files i have in a particular document and looking for the right syntax to convert all odts to pdfs. Thanks! Jake ...

How to quickly do a sitewide batch replace of CAPS HTML tags into lowercase

I have a whole bunch of HTML code throughout pages on my website and they've got all the tags and their attributes in CAPS which I don't like aesthetically, and there are a bunch of other inconsistencies and poor coding. Is there a tool or macro of some kind that can convert all the tags sitewide relatively simply without manually steppi...

Create an Array containing filenames in a directory

Hi, I'm trying to copy one or two specific files from a bunch of directories (hence why I don't want to/can't use *) from one directory to another using a batch script. Basically I want to navigate into a "root directory" and from that list all the sub-directories using dir /AD-H /B then I want to cd into each of those directories and...

processing large number of files - any library or utility to help?

I'd like to process a large number of files - think video transcoding, like youtube, where you have to transform everything into FLV format or image processing where you create thumbs from large RAW files. Is there any service or library that can help on such processing? I don't mean actual transcoding, but organizing, launching the tas...

What dummy X server should I use with openoffice on debian?

Hi, I need to run openoffice in batch mode to turn .doc files into pdf, but openoffice won't start because it needs an X server. My linux box doesn't have one. Is there some dummy blackhole X server around I can use? I'm using debian etch. ...

Sending a large request to a service in smaller batches (Java)

I work with a large Java web application which uses a service to return information. My key goal is to retrieve 100+ individual results as quick as possible. We've noticed that sending 100+ requests for information doesn't give us the best performance on the replies of data. Our solution was to break the 100+ requests into small batch...

Appending a txt file from multiple CSVs in subdirectories

Hi, I am trying to write a batch file which will append all *.csv files in the immediate subdirectories to a single text file in the current directory. From various sources I have managed to piece together this code which works fine for files in the current dir but not sub-dirs for %%a in (*.csv) do (type %%a >> csvreport.txt) If an...

Batch commit on large INSERT operation in native SQL?

I have a couple large tables (188m and 144m rows) I need to populate from views, but each view contains a few hundred million rows (pulling together pseudo-dimensionally modelled data into a flat form). The keys on each table are over 50 composite bytes of columns. If the data was in tables, I could always think about using sp_rename t...

Bash script to edit a bunch of files

To process a bunch of data and get it ready to be inserted into our database, we generate a bunch of shell scripts. Each of them has about 15 lines, one for each table that the data is going. One a recent import batch, some of the import files failed going into one particular table. So, I have a bunch of shell scripts (about 600) wher...

Batch Process while simultaneous transactions occur

I'm creating a batch process to move some information from my database to external files. This task is manageable, but there's a lot of data that needs to be processed and it will probably take about a month. During this month new information will be continuously uploaded. Is there any way I can do my batch processing and then come bac...

Caching of Map applications in Hadoop MapReduce?

Looking at the combination of MapReduce and HBase from a data-flow perspective, my problem seems to fit. I have a large set of documents which I want to Map, Combine and Reduce. My previous SQL implementation was to split the task into batch operations, cumulatively storing what would be the result of the Map into table and then performi...

Determining system requirements (hardware, processor & memory) for a batch based software application

Hi, I am tasked with building an application wherein the business users will be defining a number of rules for data manipulation & processing (e.g. taking one numerical value and splitting it equally amongst a number of records selected on the basis of the condition specified in the rule). On a monthly basis, a batch application has ...

Is it possible to have a stateless timed function.

I'm trying to set a reminder in a system to fire at a certain time. This is a web based app, so it's not like it will be in memory all the time. Ideally I'd like to avoid using a service or job on the server(mainly out of curiosity, to see if there is a more efficient way to do it) For example, imagine how many Ebay bids are constantl...

Hashing Multiple Files

Problem Specification: Given a directory, I want to iterate through the directory and its non-hidden sub-directories,  and add a whirlpool hash into the non-hidden file's names. If the script is re-run it would would replace an old hash with a new one. <filename>.<extension>   ==>  <filename>.<a-whirlpool-hash>.<e...

How to delete multiple db entities with Nhibernate?

What is the best practice for this problem? Is there any batching features built-in? Sample code: using (ITransaction transaction = _session.BeginTransaction()) { _session.Delete("FROM myObject o WHERE o.Id = IN(1,2,...99999)"); transaction.Commit(); } Thanks in advance. ...

Batch Processing Image Files in MATLAB

Hello, I'm a beginner in MATLAB and image processing. I encountered a problem while trying to use the batch processing and hope someone would be able to enlighten me. Thanks. Following the example from MATLAB, I did these: p = which('Picture1.tif'); filelist = dir([fileparts(p) filesep 'Picture*.tif']); fileNames = {filelist.name}' ...

What ways exist to distribute asynchronous batch tasks?

I am currently investigating what Java compatible solutions exist to address my requirements as follows: Timer based / Schedulable tasks to batch process Distributed, and by that providing the ability to scale horizontally Resilience, no SPFs please The nature of these tasks (heavy XML generation, and the delivery to web based recei...

Decoding File Format (Norton .FBF)

Hey there, I currently have a norton ghost backup of the 'My Documents' folder, however norton ghost does not allow me to restore 'all' my files at once, and only has the functionality to search for specific files and restore them. This is a problem as I have nearly 100GB of important documents and such, that are locked away in these .f...

WCF: Efficiently consuming large numbers of singleton requests via SQL job?

I'm planning to build a console app to run as part of a SQL 2005 job which will gather records from a database table, create a request object for a WCF service, pass this object to the service for processing, receive a response object, and update a log table with its data. This will be for processing at least several thousand records eac...