ansaurus

Question

Answer 1

+3 A:

You can surely build your own, given some statistics about letter frequencies, digraph frequencies, etc, of your target languages.

Then release it as open source. And voila, you have an open source engine for detecting the language of text!

Dolph 2010-07-03 22:10:30

Answer 2

+3 A:

I don't think you need anything very sophisticated - for example to detect if a document is in English, with a pretty high level of certainty, simply test if it contains the N most common English words - something like:

"the a an is to are in on in it"

If it contains all of those, I would say it is almost definitely English.

anon 2010-07-03 22:12:08

Unless you check most of those, there will be a risk for false positives... E.g. "Kan jag komma in och få lite is till min läsk?" would be flagged as English.

Gert G 2010-07-03 22:47:43

@Gert That's why I said "all" - of course you could also produce a percentage score. And there will always be false positives, whatever you do.

anon 2010-07-03 23:19:05

@Neil Butterworth - No problem. I understand what you mean. It's just that you have to be careful, since languages share some common elements. :)

Gert G 2010-07-03 23:30:11

I'm not looking for English in particular, my initial task is to identify which European language, including greek and swedish

niklassaers 2010-07-04 06:44:49

Answer 3

+5 A:

Depending on what you're doing, you might want to check out the python Natural Language Processing Toolkit (NLTK), which has some support for Bayesian Learning Algorithms.

In general, the letter and word frequencies would probably be the fastest evaluation, but the NLTK (or a bayesian learning algorithm in general) will probably be useful if you need to do anything beyond identification of the language. Bayesian methods will probably be useful also if you discover the first two methods have too high of an error rate.

archgoon 2010-07-03 22:39:13

Thanks for the tip, that sounds very promising. :-) Also very cool that it comes with many corpa of text, that way I won't have to train it all myself

niklassaers 2010-07-04 06:46:59

Answer 4

+3 A:

For future reference, the engine I ended up using is libtextcat which is under BSD license but seems not to be maintained since 2003. Still, it does a good job and integrates easily in my toolchain

niklassaers 2010-07-06 08:45:39

"It was primarily developed for language guessing, a task on which it is known to perform with near-perfect accuracy"

Nicolas Raoul 2010-09-27 09:41:47

http://software.wise-guys.nl/libtextcat/

Nicolas Raoul 2010-09-27 09:42:14

That's frightfully optimistic. ;-) With ~1.700.000 pages run, it detects the language correctly for ~50%, has multiple suggestions in where the language appears in ~20% more, and misses the rest. For the misses, I am lucky enough to have other data to back me up :-)

niklassaers 2010-09-27 19:49:54

ansaurus

tags:

views:

answers:

How to detect language

related questions