Ideas for converting straight quotes to curly quotes

Question

I have a file that contains "straight" (normal, ASCII) quotes, and I'm trying to convert them to real quotation mark glyphs (“curly” quotes, U+2018 to U+201D). Since the transformation from two different quote characters into a single one has been lossy in the first place, obviously there is no way to automatically perform this conversion; nevertheless I suspect a few heuristics will cover most cases. So the plan is a script (in Emacs) that does something like the following: for each straight quote character,

1. guess which curly quote character to use, if possible
2. ask the user (me) to confirm, or make a choice

This question is about the first step: what would be a good algorithm (a set of heuristics, more like) to use, for normal English text (a novel, for example)? Here are some preliminary ideas, which I believe work for double-quotes (counterexamples are welcome!):

1. If a double-quote is at the beginning of a line, guess that it is an opening quote.
2. If a double-quote is at the end of a line, guess a closing quote.
3. If a double-quote is preceded by a space, guess an opening quote.
4. If a double-quote is followed by a space, guess a closing quote.
5. If a double-quote doesn't fit into one of the above categories, guess that it is the “opposite” of the most recently used kind of double-quote.

Single quotes are trickier, because a ' might be either an opening quote, closing quote, or apostrophe, and we want to leave apostrophes alone (mustn't write “mustn’t”). Some of the same rules as above apply, but 'tis possible apostrophes are at the beginning of words (or lines), although it's less common than 'twas in the past. I can't offhand think of rules that would properly handle fragments like ["I like 'That '70s show'", she said]. It might require looking at more than just neighbouring characters, and compute distances between quotes, for example…

Any more ideas? It is okay if not all possible cases are covered; the goal is to be as intelligent as possible but no further. :-)

Edit: Some more things that might be worth thinking about (or might be irrelevant, not sure):

• quotes might not always be in matching pairs: For single quotes it's obvious why as above. But even for double quotes, when there is a quotation that extends for more than one paragraph, usual typographic convention (don't ask me why) is to start each paragraph with a quotation mark, even though it has not been closed in the previous one. So simply keeping a state machine that alternates between two states will not work!
• Nested quotation (alluded to in the "I like 'That '70s show'" example above): this might make either kind of quote not be preceded or followed by a space.
• British/American punctuation style: are commas inside the quotes or outside?
• Many word processors (e.g Microsoft Word) already do some sort of conversion like this. Although they are not perfect and can often be annoying, it might be instructive to learn how they work...

Answer 1

1. The basic thing is to always try to find matching pairs. Given that every quote has a matching quote you could make your program ask for your help only where it's unsure which is the matching quote.

2. Opening quotes are always at the opening of a line or have a space in front of them. Closing quotes always a space after them. If you find a colon with a following quote it's probably a closing quote.

3. If the letter following the quote is upper case it's probably an opening quote.

4. If there's a punctuation mark in front of the quote it's probably a closing quote.

5. Try to do it iteratively. The program should ask you first for all the quotes that it can definitely assign to a function. (Just to make sure it hasn't made any errors.)

6. In the second round something like all the quotes that it's unsure whether they are opening quotes or apostrophes. For all opening quotes it has to find automatically the closing quote.

Another, maybe less complex, idea could be:

1. Find all non-quotes by asking the user about each one that could potentially be a quote or a non-quote.

2. All the remaining quotes should be fairly easy to convert. Opening quotes have a spaces or newline in front of them and closing after them.

One last piece of thought:

You should break the process apart like processing only paragraph-wise. If your program makes an error, which it probably will given the complexity of language, it's easier for you to correct it and the program can start fresh with the new paragraph.

Answer 2

A good place to start would be with a state machine:

• Starting at position 0, iterate over the characters
• Upon finding a quote, enter the "Quoted" state ( open quote )
• If in "Quoted" state and you encounter a quote, return to "Starting" state ( closing quote )

You can make additional decisions at each of the state transitions.

You could attempt to normalize the single quotes by identifying known conjunctions, for instance, and converting them to a different, not text, character prior to processing.

My $0.02

Answer 3

guess which curly quote character to use, if possible

It is not, in the general case.

The simple algorithm that most automatic converters use is just to look at the previous letter you typed before the ' or ". If it's a space, start of line, opening bracket or other opening quote, choose opening quote, else closing. The advantage of this method is that it can run as-you-type, so when it chooses the wrong one you can generally correct it.

we want to leave apostrophes alone

I agree! But not many people do. It's normal typesetting practice to turn an apostrophe into a left-facing single quote. Personally I prefer to leave them as they are, to distinguish them from enclosing quotes, making the text easier (I find) to read, and possible to process automatically.

However this really is just my taste and is not generally considered justified merely because the character is defined by the Unicode standard as being APOSTROPHE.

'tis possible apostrophes are at the beginning of words

Indeed. There is no way to tell an apostrophe from a potential open quote in cases like the classic Fish 'n' Chips, short of enormous amounts of cultural context.

(Not to mention primes, okinas, glottal stops and various other uses of the apostrophe...)

The best thing to do, of course, is install a keyboard layout that can type smart quotes directly. I have ‘’ on AltGr+[], “” on AltGr+Shift+[], –— on AltGr+[Shift]+dash, and so on.

Answer 4

I hate to say it, but the best course of action might be to study what Word does, and copy it. Even if it's wrong in some cases, it represents a standard that many people have become accustomed to. One behavior to emulate is having undo (Ctrl-Z) immediately revert to the straight quote after you have substituted a curved one.

Answer 5

["I like 'That '70s show'", she said]

I originally thought maybe using multiple passes over the text to gain context insight might help but that would not solve all instances.

The best thing you could do is run up a list of possible word sets/expressions like 'twas, 'tis, '70's etc. and throw them in the dictionary with auto-correction on them to convert the straights to curls and vice versa. Spell checks run on every word anyway don't they? (sorry that doesn't help your emacs problem)

OO ignores the single quote curving all together from what I can tell.

Wikipedia has a bit of info on these pesky things.

Answer 6

It looks like your initial post covers most of the ideas I was going to write here, this is what I've got left...

For the apostrophe example ("I like 'That '70s show'", she said), it's unlikely that quotes will be nested directly inside quotes of the same type. You could take advantage of that.

Best way to do this in my opinion is to make the code only handle unambiguous cases (double quotes are pretty simple). For the ones with multiple possible choices, store their position in a list and examine it when it's finished. You might find a few more easily-coded cases in there, or you might just decide to fix them manually.

Answer 7

Here is a regular expression that might help for double-quotes:

/([^\s\(]?)"(\s*)([^\\]*?(\\.[^\\]*)*)(\s*)("|\n\n)([^\s\)\.\,;]?)/gms

It will restart at each paragraph, and it will identify pairs of quotes (and will also allow you to check that the spacing is correct before and after the quotes, if that's useful).

Numbered element    identification  
  1               non-white-space before quote quote  
  2               white-space after leading quote  
  5               white-space before trailing quote  
  6               trailing quote (or double-newline, i.e. start of a paragraph  
  7               character after trailing quote if not whitespace or right   paren

I think it would be reasonable to extend this for your other cases (I just haven't had the need to yet.)

It's javascript syntax. It's pretty fast, but I haven't done more optimizing than my "good enough". It will do a, say, 400 page book in about a second. I think it would be hard to match its speed procedurally.

Answer 8

Computational linguistics anyone?

Somebody mentioned if you had a vast amount of cultural context, it might be feasible. So the overkill but most accurate automated solution to the problem is shallow parsing. This requires a corpus of whatever language and mode you're dealing with (e.g. the Brown corpus for general English).

Develop a classifier for curly quotes based on the syntactic context of the curly quotes occurring in the corpus. Finally, give your arbitrary syntactic context with a straight quote to your classifier and out pops the most probable quote character!