Automating an application

Question

I've always wondered the best way to automate use of a GUI in windows. When I was about 15, I wrote a little application that used some simple Windows API functions to automatically click on certain locations on the screen based on a script. This could be used to automate GUI applications, but surely it's not the best way.

So, my question is: What's the best way to automate use of a GUI in windows? Are there certain windows API functions that would be beneficial? If the program were to crash, how could you detect that?

Answer 1

Take a look at the SendMessage function. It essentially does the same thing as your script did, except it would send the clicks directly to the button object. The function is:

LRESULT SendMessage(      
    HWND hWnd,
    UINT Msg,
    WPARAM wParam,
    LPARAM lParam
);

You would do something like:

SendMessage([your_window_hwnd], WM_LBUTTONDOWN, 0, y << 16 | x);
SendMessage([your_window_hwnd], WM_LBUTTONUP, 0, y << 16 | x);

You can also send keyboard presses with:

SendMessage([your_window_hwnd], WM_KEYDOWN, [key_value], 0);
SendMessage([your_window_hwnd], WM_KEYUP, [key_value], 0);

Answer 2

Have a look at Project SIKULI:

Sikuli is a visual technology to search and automate graphical user interfaces (GUI) using images (screenshots). (...)

Answer 3

To determine if the program is still running, GetProcessExitCode. For event-driven notification of crashes, use any of the wait functions with a process handle with SYNCHRONIZE access (If you know the PID, use OpenProcess to get a HANDLE).

Then use Spy++ to find out things like control child IDs and WM_COMMAND arguments, then use PostMessage, SendMessage, SetDlgItemText, and SendDlgItemMessage. You'll probably be mostly sending WM_COMMAND (menu items and toolbars), WM_NOTIFY (inform dialog about interaction with child controls, like buttons being pressed or textbox text changing), and may have to send messages directly to child controls to e.g. select an item within a listview or set the text of an editbox.

As long as the program supports cut+paste, it's unlikely you'll actually have to synthesize individual keystrokes.

Answer 4

The best way to automate a GUI is to control it without using low-level calls. Using low-level calls is more complicated and more fragile than automation needs to be. The best way to automate a GUI is to use a tool designed for that purpose. AutoIt is a good free tool. White is a free library for UI automation. There are other options, like Sikuli, but those two are usually my first recommendations.

If you want to automate for testing purposes, things get more complicated.