What is the best, python or bash for selectively concatenating lots of files?

Question

Hi,

I have around 20000 files coming from the output of some program, and their names follow the format:

data1.txt
data2.txt
...
data99.txt
data100.txt
...
data999.txt
data1000.txt
...
data20000.txt

I would like to write a script that gets as input argument the number N. Then it makes blocks of N concatenated files, so if N=5, it would make the following new files:

data_new_1.txt: it would contain (concatenated) data1.txt to data5.txt (like cat data1.txt data2.txt ...> data_new_1.txt )

data_new_2.txt: it would contain (concatenated) data6.txt to data10.txt
.....

I wonder what do you think would be the best approach to do this, whether bash, python or another one like awk, perl, etc.

The best approach I mean in terms of simplest code.

Thanks

Answer 1

Since this can easily be done in any shell I would simply use that.

This should do it:

#!/bin/sh
FILES=$1
FILENO=1

for i in data[0-9]*.txt; do
    FILES=`expr $FILES - 1`
    if [ $FILES -eq 0 ]; then
        FILENO=`expr $FILENO + 1`
        FILES=$1
    fi

    cat $i >> "data_new_${FILENO}.txt"
done

Python version:

#!/usr/bin/env python

import os
import sys

if __name__ == '__main__':
    files_per_file = int(sys.argv[1])

    i = 0
    while True:
        i += 1
        source_file = 'data%d.txt' % i
        if os.path.isfile(source_file):
            dest_file = 'data_new_%d.txt' % ((i / files_per_file) + 1)
            file(dest_file, 'wa').write(file(source_file).read())
        else:
            break

Answer 2

Best in what sense? Bash can do this quite well, but it may be harder for you to write a good bash script if you are more familiar with another scripting language. Do you want to optimize for something specific?

That said, here's a bash implementation:

 declare blocksize=5
 declare i=1
 declare blockstart=1
 declare blockend=$blocksize
 declare -a fileset 
 while [ -f data${i}.txt ] ; do
         fileset=("${fileset[@]}" $data${i}.txt)
         i=$(($i + 1))
         if [ $i -gt $blockend ] ; then
                  cat "${fileset[@]}" > data_new_${blockstart}.txt
                  fileset=() # clear
                  blockstart=$(($blockstart + $blocksize))
                  blockend=$(($blockend+ $blocksize))
         fi
 done

EDIT: I see you now say "Best" == "Simplest code", but what's simple depends on you. For me Perl is simpler than Python, for some Awk is simpler than bash. It depends on what you know best.

EDIT again: inspired by dtmilano, I've changed mine to use cat once per blocksize, so now cat will be called 'only' 4000 times.

Answer 3

Here's a Python (2.6) version (if you have Python 2.5, add a first line that says

from __future__ import with_statement

and the script will also work)...:

import sys

def main(N):
   rN = range(N)
   for iout, iin in enumerate(xrange(1, 99999, N)):
       with open('data_new_%s.txt' % (iout+1), 'w') as out:
           for di in rN:
               try: fin = open('data%s.txt' % (iin + di), 'r')
               except IOError: return
               out.write(fin.read())
               fin.close()

if __name__ == '__main__':
    if len(sys.argv) > 1:
        N = int(sys.argv[1])
    else:
        N = 5
    main(N)

As you see from other answers & comments, opinions on performance differ -- some believe that the Python startup (and imports of modules) will make this slower than bash (but the import part at least is bogus: sys, the only needed module, is a built-in module, requires no "loading" and therefore basically negligible overhead to import it); I suspect avoiding the repeated fork/exec of cat may slow bash down; others think that I/O will dominate anyway, making the two solutions equivalent. You'll have to benchmark with your own files, on your own system, to solve this performance doubt.

Answer 4

Let's say, if you have a simple script that concatenates files and keeps a counter for you, like the following:

#!/usr/bin/bash
COUNT=0
if [ -f counter ]; then
  COUNT=`cat counter`
fi
COUNT=$[$COUNT+1]
echo $COUNT > counter
cat $@ > $COUNT.data

The a command line will do:

find -name "*" -type f -print0 | xargs -0 -n 5 path_to_the_script

Answer 5

I like this one which saves on executing processes, only 1 cat per block

#! /bin/bash

N=5 # block size
S=1 # start
E=20000 # end

for n in $(seq $S $N $E)
do
    CMD="cat "
    i=$n
    while [ $i -lt $((n + N)) ]
    do
        CMD+="data$((i++)).txt "
    done
    $CMD > data_new_$((n / N + 1)).txt
done

Answer 6

Simple enough?

make_cat.py

limit = 1000
n = 5
for i in xrange( 0, (limit+n-1)//n ):
     names = [ "data{0}.txt".format(j) for j in range(i*n,i*n+n) ]
     print "cat {0} >data_new_{1}.txt".format( " ".join(names), i )

Script

python make_cat.py | sh

Answer 7

how about a one liner ? :)

ls data[0-9]*txt|sort -nk1.5|awk 'BEGIN{rn=5;i=1}{while((getline _<$0)>0){print _ >"data_new_"i".txt"}close($0)}NR%rn==0{i++}'