Opennlp 1.5 for SentenceDetector?

Question

Now I have the following code:

SentenceModel sd_model = null;
  try {
   sd_model = new SentenceModel(new FileInputStream(
     "opennlp/models/english/sentdetect/en-sent.bin"));
  } catch (InvalidFormatException e) {
   // TODO Auto-generated catch block
   e.printStackTrace();
  } catch (FileNotFoundException e) {
   // TODO Auto-generated catch block
   e.printStackTrace();
  } catch (IOException e) {
   // TODO Auto-generated catch block
   e.printStackTrace();
  }
  SentenceDetectorME mSD = new SentenceDetectorME(sd_model);
  String param = "This is a good senttence.I'm very happy. Who can tell me the truth.And go to school.";
  String[] sents = mSD.sentDetect(param);
  for(String sent : sents){
   System.out.println(sent);
  }

But I got the follwing results:

This is a good senttence.I'm very happy.
Who can tell me the truth.And go to school.

Absolutely, this isn't what we want. How can I fix the problem? thanx.

Answer 1

Try using the language specific sentence detector (opennlp.tools.lang.english.SentenceDetector).

Answer 2

I don't think the sentence detection model provided with OpenNLP is a good fit for your task because it has been trained on data where whitespace follows sentence-final punctuation, since this is fairly standard in English orthography. English sentence detectors are typically intended to distinguish between sentence-final punctuation and punctuation used mid-sentence in abbreviations, quotes, etc. In all cases, your run-of-the-mill sentence detector is going to expect some kind of whitespace between sentences.

If you want to use OpenNLP, I think the easiest solution would be to preprocess your data to add a space where you detect a pattern like [a-z][.?!][A-Z]. (This pattern clearly isn't sufficient, but just to give an idea.) There aren't many abbreviations that have formats like Nnnn.Nnnn or Nnnn?Nnnn so I bet you could achieve good results without using anything fancier than a regular expression, but that would depend on what your data looks like. Alternatively, you could use some kind of tokenizer with a custom model to find these cases.

It's also possible you could train your own sentence detection model that doesn't expect whitespace between sentences, but it looks like that's going to be tricky with OpenNLP. Their provided training programs expect training data with one sentence per line, so there's no way to avoid inserting whitespace between sentences.