How to get text from the screen

Question

There is some Win OS API call or so that would let one obtain text from the screen

not via obtaining a snapshot and then doing OCR on it, but via API

the idea is to get the text that is under the mouse that the user points to and clicks on.

This is how tools like Babylon ( www.babylon.com ) and 1-Click Answers ( http://www.answers.com/main/download_answers_win.jsp ) and many others work.

Can someone point me to the right direction(s) to get this functionality?

Answer 1

i think its called the clipboard. i am going to bet these programs inject click and double click & keyboard events and then copy items there for inspection. Alternatively, they are gettin jiggy with the windows text controls, and grabbing content that way. i suspect due to security issues, these tools have problems running in vista also.

Answer 2

There is no direct way to obtain text. An application could render text in a zillion different ways (Windows API being one of them), and after it's rendered - it's just a bunch of pixels.

A method you could try however is to find the window directly under the mouse and trying to get the text from them. This would work fine on most standard Windows controls (labels, textboxes, etc.) Wouldn't work on Internet browsers though.

I think the best you can do is make your application such that it supports as many different (common) controls as possible in the above described manner.

Answer 3

You can get the text of every window with the GetWindowText API. The mouse position can be found with the GetCursorPos API.

In Delphi you could use this function (kudos to Peter Below)

Function ChildWindowUnderCursor: HWND;
Var
  hw, lasthw: HWND;
  pt, clientpt: TPoint;
Begin
  Result := 0;
  GetCursorPos( pt );
  // find top-level window under cursor
  hw := WindowFromPoint( pt );
  If hw = 0 Then Exit;

  // look for child windows in the window recursively
  // until we find no new windows
  Repeat
    lasthw := hw;
    clientpt := Pt;
    Windows.ScreenToClient( lasthw, clientpt );
    // Use ChildwindowfromPoint if app needs to run on NT 3.51!
    hw := ChildwindowFromPointEx( lasthw, clientpt, CWP_SKIPINVISIBLE );
  Until hw = lasthw;
  Result := hw;
End;

Regards,
Lieven