How do I track an animated object in Python?

Question

I want to automate playing a video game with Python. I want to write a script that can grab the screen image, diff it with the next frame and track an object to click on. What libraries would be useful for this other than PIL?

Answer 1

There are a few options here. The brute force diff'ing approach will lead to a lot of frustration unless what you're tracking is very consistent. For this you could use any number of genetic approaches to train your program what to follow. After enough generations it would do the right thing reliably. If the thing you want to track is visually obvious (like a red ball on a white screen) then you could detect it yourself through simple brute force scanning of the bitmap.

Another approach would be just looking at the memory of the running app, and figuring out what area is controlling the position of your object. For some more info and ideas on this, see how mumble got 3D positional audio working in various games.

http://mumble.sourceforge.net/HackPositionalAudio

Answer 2

Answer would depend on the platform and game too. e.g. I did once similar things for helicopter flash game, as it was very simple 2d game with well defined colored maze It was on widows with copy to clipboard and win32 key events using win32api bindings for python.