Python - lines from files - all combinations

Question

I have two files - prefix.txt and terms.txt both have about 100 lines. I'd like to write out a third file with the Cartesian product

http://en.wikipedia.org/wiki/Join_(SQL)#Cross_join

-about 10000 lines.

What is the best way to approach this in Python?

Secondly, is there a way to write the 10,000 lines to the third file in a random order?

Answer 1

A Cartesian product enumerates all combinations. The easiest way to enumerate all combinations is to use nested loops.

You cannot write files in a random order very easily. To write to a "random" position, you must use file.seek(). How will you know what position to which you will seek? How do you know how long each part (prefix+term) will be?

You can, however, read entire files into memory (100 lines is nothing) and process the in-memory collections in "random" orders. This will assure that the output is randomized.

Answer 2

from random import shuffle
a = list(open('prefix.txt'))
b = list(open('terms.txt'))
c = [x.strip() + y.strip() for x in a for y in b]
shuffle(c)
open('result.txt', 'w').write('\n'.join(c))

Certainly, not the best way in terms of speed and memory, but 10000 is not big enough to sacrifice brevity anyway. You should normally close your file objects and you can loop through at least one of the files without storing its content in RAM. This: [:-1] removes the trailing newlline from each element of a and b.

Edit: using s.strip() instead of s[:-1] to get rid of the newlines---it's more portable.

Answer 3

You need itertools.product.

for prefix, term in itertools.product(open('prefix.txt'), open('terms.txt')):
    print(prefix.strip() + term.strip())

Print them, or accumulate them, or write them directly. You need the .strip() because of the newline that comes with each of them.

Afterwards, you can shuffle them using random.shuffle(list(open('thirdfile.txt')), but I don't know how fast that will be on a file of the sizes you are using.