Screen scraping an application window and interacting with the mouse and keyboard

Question

The other day I found myself addicted to a flash game and frustrated by the thing at the same time. In a moment of frustration with the game I thought I would make a 'bot' to beat it for me. Well, I really wouldn't, but it made me realize: I don't know how to interact with another application in a way to do this. Which brings me to the question, how would one take screenshots of another running application and interact with it with the keyboard and mouse. Ideally the solution would be in a managed language like c#.

When doing the background reading the net was drowning with articles on scrapping HTML. There were not many articles on actually screen scrapping an application.

Diverse answers are appreciated as I’m really looking at surveying what’s out there.

UPDATE

I'm looking for a way to interface with another application rather than script/macro another application.

UPDATE

Could something like Xming be used to redirect the interface? http://www.straightrunning.com/XmingNotes/

Perhaps a Terminal Services client? http://www.codeproject.com/KB/cs/RemoteDesktop_CSharpNET.aspx

Answer 1

Check out Sikuli, it is basically what you are looking for. It is written in Java however.

http://groups.csail.mit.edu/uid/sikuli/

Answer 2

I have used AutoHotKey for application automation.

Answer 3

I ended up creating the bot as a recreational coding project. For screen scrapping, a Graphics handler can simply be used to 'CopyFromScreen' into a Bitmap. I located the flash region of the screen using a Matlab script to subtract a grayscale screenshot of the game from the entire screenshot where the subtraction led to the least intense image (optimization toolbox compiled into a .NET library). Once the location was known I used AForge.NET to parse the game state from the game's screenshot. I took a screenshot of each game piece and gathered the Histogram stats (avoiding OCR). Then the Histograms are calculated for each piece's location on the board and the closest match is taken by using a Root Sum Square (RSS) on the RGB mean values. This provides a reliable game state. From there I used user32.dll interop to send mouse commands to the respective locations to perform the moves. There were some tricks regarding the solver for the game, but thats not relevant to this post. Sadly the first run blew away my and any of my friend's scores at 765,000 (Bejeweled Blitz).