The idea of text highlighting, code completion, etc in programming

Question

Hello guys, I wanna know the idea of advanced text editors features like text highlighting, code completion, automatic indentation etc.

To make my idea clear I imagine that text highlighting is reading the entire text into a string then do regular expression replacement of keywords with keywords + color codes and replace the text again. That looks logical but it would be so inefficient to do that with every keystroke when your file is 4000 lines for example ! So I wanna know the idea of implementation of such thing in C# for example (any other language would be fine also but that's what i am experimenting with right now).

Answer 1

Syntax highlighting: This comes to mind. I haven't actually tried the example, so I cannot say anything about the performance, but it seems to be the simplest way of getting basic syntax highlighting up and running.

Auto-completion: Given a list of possible keywords (that could be filtered based on context) you could quickly discard anything that doesn't match what the user is currently typing. In most languages, you can safely limit yourself to one "word", since whitespace isn't usually legal in an identifier. For example, if I start typing "li", the auto-completion database can discard anything that doesn't start with the letters 'l' and 'i' (ignoring case). As the user continues to type, more and more options can be discarded until only one -- or at least a few -- remains. Since you're just looking at one word at a time, this would be very fast indeed.

Indentation: A quick-and-dirty approach that would (kind of) work in C-like languages is to have a counter that you increment once for every '{', and decrement once for every '}'. When you hit enter to start a new line, the indentation level is then counter * indentWidth, where indentWidth is a constant number of spaces or tabs to indent by. This suffers from a serious drawback, though -- consider the following:

if(foo)
    bar(); // This line should be indented, but how does the computer know?

To deal with this, you might look for lines that end with a ')', not a semicolon.

Answer 2

An old, but still applicable resource for editor internals is The Craft of Text Editing. Chapter 7 address the question of redisplay strategies directly.

Answer 3

In order to do "advanced" syntax highlighting -- that is, highlighting that requires contextual knowledge, a parser is often needed. Most parsers are built on some sort of a formal grammar which exist in various varieties: LL, LALR, and LR are common.

However, most parsers operate on whole files, which is quite inefficient for text editing, so instead we turn to incremental parsers. Incremental parsers use knowledge of the language and the structure of what has been previously been processed in order to re-do the least amount of work possible.

Here's a few references to incremental parsing:

• Language Design Patterns and incremental parsing
• Incremental Parsing
• Incremental Parsing in the Yi Editor and the presentation on Vimeo