Natural language parsing, practical example

Question

I am looking to use a natural language parsing library for a simple chat bot. I can get the Parts of Speech tags, but I always wonder. What do you do with the POS. If I know the parts of the speech, what then?

I guess it would help with the responses. But what data structures and architecture could I use.

Answer 1

Natural language processing is wide and deep, with roots going back at least to the 60s. You could start reading up on computational linguistics in general, natural language generation, generative grammars, Markov chains, chatterbots and so forth.

Wikipedia has a short list of libraries which I assume you might have seen. Java doesn't have a long tradition in NLP, though I haven't looked at the Stanford libraries.

I doubt you'll get very impressive results without diving fairly deeply into linguistics and grammar. Not everybody's favourite school subject (or so I've heard reported -- loved'em meself!).

Answer 2

I'll skip a lot many details and keep this simple. Parts of Speech tagging help you to create an parse tree out of a sentence. Once you have this, you try to make out a meaning as unambiguously as possible. The result of this parsing step will greatly aid you to frame a suitable response for you chatterbot.

Answer 3

A part-of-speech tagger assigns labels to the words in the input text. For example, the popular Penn Treebank tagset has some 40 labels, such as "plural noun", "comparative adjective", "past tense verb", etc. The tagger also resolves some ambiguity. For example, many English word forms can be either nouns or verbs, but in the context of other words, their part of speech is unambiguous. So, having annotated your text with POS tags you can answer questions like: how many nouns do I have?, how many sentences do not contain a verb?, etc.

For a chatbot, you obviously need much more than that. You need to figure out the subjects and objects in the text, and which verb (predicate) they attach to; you need to resolve anaphors (which individual does a he or she point to), what is the scope of negation and quantifiers (e.g. every, more than 3), etc.

Ideally, you need to map you input text into some logical representation (such as first-order logic), which would let you bring in reasoning to determine if two sentences are equivalent in meaning, or in an entailment relationship, etc.

While a POS-tagger would map the sentence

Mary likes no man who owns a cat.

to such a structure

Mary/NNP likes/VBZ no/DT man/NN who/WP owns/VBZ a/DT cat/NN ./.

you would rather need something like this:

SubClassOf(
   ObjectIntersectionOf(
      Class(:man)
      ObjectSomeValuesFrom(
         ObjectProperty(:own)
         Class(:cat)
      )
   )
   ObjectComplementOf(
      ObjectSomeValuesFrom(
         ObjectInverseOf(ObjectProperty(:like))
         ObjectOneOf(
            NamedIndividual(:Mary)
         )
      )
   )
)

Of course, while POS-taggers get precision and recall values close to 100%, more complex automatic processing will perform much worse.

A good Java library for NLP is LingPipe. It doesn't, however, go much beyond POS-tagging, chunking, and named entity recognition.

Answer 4

Once you have part of speech tags you can extract, for example, all nouns, so you know roughly what things or objects someone is talking about.

To give you an example:

Someone says "you can open a new window." When you have the POS tags you know they are not talking about a can (as in container, jar etc., which would even make sense in the context of open), but a window. You'll also know that open is a verb.

With this information, your chat bot can generate a much better reply that will have nothing to do with can openers etc.

Note: You don't need a parser to get POS tags. A simple POS tagger is enough. A parser will give you even more information (e.g. what is the subject, what the object of the sentence?)