views:

87

answers:

3

Hi,

I'm designing architecture of a text parser. Example sentence: Content here, content here.

Whole sentence is a... sentence, that's obvious. The, quick etc are words; , and . are punctuation marks. But what are words and punctuation marks all together in general? Are they just symbols? I simply don't know how to name what a single sentence consists of in the most reasonable abstract way (because one may write it consists of letters/vowels etc).

Thanks for any help :)

+2  A: 

A common term comprising the two sub-categories "words" and "punctuation", often used when talking about parsing, is "tokens".

Alex Martelli
Sentence is also a token in this manner, I think. I don't want to refer to parsing terminology, but to the real life one :)
shazarre
A sentence is never a token, it is composed of tokens. The parser composes the tokens of a sentence into a parse tree. If you want a real life (assuming you mean non-technical) term, it will be a little rough as people don't feel the need to combine words and punctuation into a singular term that can represent an instance of either.
jball
+2  A: 

Depending on what stage of your lexical analysis of input text you are looking at, these would be either "lexemes" or "tokens."

Curt Sampson
+2  A: 

What you're doing is technically lexical analysis ("lexing"), which takes a sequence of input symbols and generates a series of tokens or lexemes. So word, punctuation and white-space are all tokens.

In (E)BNF terms, lexemes or tokens are synonymous with "terminal symbols". If you think of the set of parsing rules as a tree the terminal symbols are the leaves of the tree.

So what's the atom of your input? Is it a word or a sentence? If it's words (and white-space) then a sentence is more akin to a parsing rule. In fact the term "sentence" can itself be misleading. It's not uncommon to refer to the entire input sequence as a sentence.

A semi-common term for a sequence of non-white-space characters is a "textrun".

cletus
Input will be, in general, any text. It may consist of many sentences, only one or only few words. They will be then analyzed to get eventual sentences and perform further analisys. But an atom... it's not word, because there are other symbols, too. I wanted to know how to call them for simplicity of design. Thanks for your help.
shazarre