Get the word under the mouse cursor in Windows

Question

Greetings everyone,

A friend and I are discussing the possibility of a new project: A translation program that will pop up a translation whenever you hover over any word in any control, even static, non-editable ones. I know there are many browser plugins to do this sort of thing on webpages; we're thinking about how we would do it system-wide (on Windows).

Of course, the key difficulty is figuring out the word the user is hovering over. I'm aware of MSAA and Automation, but as far as I can tell, those things only allow you to get the entire contents of a control, not the specific word the mouse is over.

I stumbled upon this (proprietary) application that does pretty much exactly what we want to do: http://www.gettranslateit.com/

Somehow they are able to get the exact word the user is hovering over in almost any application (It seems to have trouble in a few apps, notably Windows Explorer). It even grabs text out of obviously custom-drawn controls, somehow. At first I thought it must be using OCR. But even when I shrink the font so far down that the text becomes a completely unreadable blob, it can still recognize words perfectly. (And yet, it doesn't recognize anything if I change the font to Wingdings. But maybe that's by design?)

Any ideas as to how it's achieving this seemingly impossible task?

EDIT: It doesn't work with Wingdings, but it does work with some other nonsense fonts, so I've confirmed it can't be OCR.

Answer 1

You could capture the GDI calls that output text to the display, and then figure out which word's bounding box the cursor falls in.

Answer 2

Well, for GDI controls you can get the position and size of the control, and you can usually get the font info. For example, with static text controls you'd use WM_GETFONT. Then once you have that you can get the position of the mouse relative to the position of the control and use one of the font functions, perhaps something like GetTextExtentPoint32 to figure out what is under the cursor. I'm pretty sure the answer lies in that direction...

You can run dumpbin /imports on the other application and see what APIs they are calling.