What's the fastest way to fixup line-endings for SMTP sending?

Question

Hello all. I'm coding a email application that produces messages for sending via SMTP. That means I need to change all lone \n and \r characters into the canonical \r\n sequence we all know and love. Here's the code I've got now:

CRLF = '\r\n'
msg = re.sub(r'(?<!\r)\n', CRLF, msg)
msg = re.sub(r'\r(?!\n)', CRLF, msg)

The problem is it's not very fast. On large messages (around 80k) it takes up nearly 30% of the time to send a message!

Can you do better? I eagerly await your Python gymnastics.

Answer 1

You could start by pre-compiling the regexes, e.g.

FIXCR = re.compile(r'\r(?!\n)')
FIXLN = re.compile(r'(?<!\r)\n')

Then use FIXCR.sub and FIXLN.sub. Next, you could try to combine the regexes into one, with a | thingy, which should also help.

Answer 2

Something like this? Compile your regex.

CRLF = '\r\n'
cr_or_lf_regex = re.compile(r'(?:(?<!\r)\n)|(?:\r(?!\n))')

Then, when you want to replace stuff use this:

cr_or_lf_regex.sub(CRLF, msg)

EDIT: Since the above is actually slower, let me take another stab at it.

last_chr = ''

def fix_crlf(input_chr):
    global last_chr
    if input_chr != '\r' and input_chr != '\n' and last_chr != '\r':
        result = input_chr
    else:
        if last_chr == '\r' and input_chr == '\n': result = '\r\n'
        elif last_chr != '\r' and input_chr == '\n': result = '\r\n'
        elif last_chr == '\r' and input_chr != '\n': result = '\r\n%s' % input_chr
        else: result = ''

    last_chr = input_chr
    return result

fixed_msg = ''.join([fix_crlf(c) for c in msg])

Answer 3

This regex helped:

re.sub(r'\r\n|\r|\n', '\r\n', msg)

But this code ended up winning:

msg.replace('\r\n','\n').replace('\r','\n').replace('\n','\r\n')

The original regexes took .6s to convert /usr/share/dict/words from \n to \r\n, the new regex took .3s, and the replace()s took .08s.

Answer 4

Replace them on the fly as you're writing the string to wherever it's going. If you use a regex or anything else you'll be making two passes: one to replace the characters and then one to write it. Deriving a new Stream class and wrapping it around whatever you're writing to is pretty effective; that's the way we do it with System.Net.Mail and that means I can use the same stream encoder for writing to both files and network streams. I'd have to see some of your code in order to give you a really good way to do this though. Also, keep in mind that the actual replacement won't really be any faster, however the total execution time would be reduced since you're only making one pass instead of two (assuming you actually are writing the output of the email somewhere).