Simple python/Regex problem: Removing all new lines from a file

Question

I'm becoming acquainted with python and am creating problems in order to help myself learn the ins and outs of the language. My next problem comes as follows:

I have copied and pasted a huge slew of text from the internet, but the copy and paste added several new lines to break up the huge string. I wish to programatically remove all of these and return the string into a giant blob of characters. This is obviously a job for regex (I think), and parsing through the file and removing all instances of the newline character sounds like it would work, but it doesn't seem to be going over all that well for me.

Is there an easy way to go about this? It seems rather simple.

Answer 1

import re
re.sub("\n", "", file-contents-here)

Answer 2

I wouldn't use a regex for simply replacing newlines - I'd use string.replace(). Here's a complete script:

f = open('input.txt')
contents = f.read()
f.close()
new_contents = contents.replace('\n', '')
f = open('output.txt', 'w')
f.write(new_contents)
f.close()

Answer 3

The two main alternatives: read everything in as a single string and remove newlines:

clean = open('thefile.txt').read().replace('\n', '')

or, read line by line, removing the newline that ends each line, and join it up again:

clean = ''.join(l[:-1] for l in open('thefile.txt'))

The former alternative is probably faster, but, as always, I strongly recommend you MEASURE speed (e.g., use python -mtimeit) in cases of your specific interest, rather than just assuming you know how performance will be. REs are probably slower, but, again: don't guess, MEASURE!

So here are some numbers for a specific text file on my laptop:

$ python -mtimeit -s"import re" "re.sub('\n','',open('AV1611Bible.txt').read())"
10 loops, best of 3: 53.9 msec per loop
$ python -mtimeit "''.join(l[:-1] for l in open('AV1611Bible.txt'))"
10 loops, best of 3: 51.3 msec per loop
$ python -mtimeit "open('AV1611Bible.txt').read().replace('\n', '')"
10 loops, best of 3: 35.1 msec per loop

The file is a version of the KJ Bible, downloaded and unzipped from here (I do think it's important to run such measurements on one easily fetched file, so others can easily reproduce them!).

Of course, a few milliseconds more or less on a file of 4.3 MB, 34,000 lines, may not matter much to you one way or another; but as the fastest approach is also the simplest one (far from an unusual occurrence, especially in Python;-), I think that's a pretty good recommendation.

Answer 4

I know this is a python learning problem, but if you're ever trying to do this from the command-line, there's no need to write a python script. Here are a couple of other ways:

cat $FILE | tr -d '\n'

awk '{printf("%s", $0)}' $FILE

Neither of these has to read the entire file into memory, so if you've got an enormous file to process, they might be better than the python solutions provided.