I need to do natural language detection (with confidence scores), preferably in Java, I'd really not introduce more platforms/technologies at this stage of the project. I have previously used the Google API for this in a PoC, but I now need to scale up to very large amounts of data, so any web-based solution won't cut it, (also Google are unclear in their TOS).
+3
A:
While it doesn't provide confidence scores, you could at least start from my cue.language library, and modify its language-detection stuff to return all of the languages it found, with a simple confidence score. (It would be an easy modification.)
Jonathan Feinberg
2009-12-17 19:02:31
Nice! This is stop-word based, yes?
johanbev
2009-12-17 19:07:31
Yes, exactly. It's very dumb, but "good enough" for my purposes!
Jonathan Feinberg
2009-12-17 19:07:57
A:
We sell one, with confidence scores. www.basistech.com. If you are an academic we can probably let you use if for free. If you need open source, of course, there are other answers being posted.
bmargulies
2009-12-17 19:03:25
+3
A:
You need to determine what language a text is in? I've used some n-gram-based algorithms for this. NGramJ is a Java-implementation. (One of several, I have no opinion on which Java-implementation is the better one)
Alex Brasetvik
2009-12-17 19:03:33
Just to clarify, this uses character n-grams, which makes decisions based on the frequencies with which letters appear next to each other. I've heard that this kind of technique should be able to work on text as short as a few words' search query.
Ken Bloom
2009-12-21 00:41:55
+2
A:
Take a look at Nutch' Language Identifier. I have used it couple of years back with excellent results.
Shashikant Kore
2009-12-18 10:26:09
Ended up using this, good tool, and pretrained too! Integration took no more than 30 min. THX mate!
johanbev
2009-12-20 15:56:10
A:
tnnrg
2010-10-15 04:45:25