chunking/text parsing using NLTK

Question

I am trying to parse some text and diagram it, like you would a sentence. I am new to NLTK and am trying to find something in NLTK that will help me accomplish this. So far, I have seen nltk.ne_chunk and nltk.pos_tag. I find them to be not very helpful and I am not able to find any good online documentation.

I have also tried to use the LancasterStemmer, but I don't fully understand what it does or how it should be used or why it even exists.

Can somebody please help me out with this? I'm really at a loss and getting quite frustrated without any guiding lights.

Thanks in advance

Answer 1

What you are describing is actually a really hard task, as in the end, whether your program has succeeded or failed is an entirely subjective measure. When this is the case, it usually means constructing a program to solve the problem is hard. There are people who get paid to work on these kind problems in universities.

If you wanted to have a stab at it, I'd suggest trying for to use some kind on automated lexical analysis tool rather than trying to manually parse and annotate, and then leverage your parse tree. Usually parse-trees represent syntactic analyses, ie the structure of the sentence. You on the other hand are concerned rather with semantic analysis, ie what it means - or at least whether two sentences are similar or different (which is actually a bit easier than what something means).

You could look into some off-the-shelf automatic summarization tools. These try to score sentences by how important they are to a piece of text and filter out sentences which are less important than a specified threshold. Not that this really helps you that much as you still have the problem of needing the merge the summaries.