We need to automate GUI testing of an application developed in Win32 API. Developer's have created this application by custom painted controls. They have controls which look like Grid, Buttons etc., but they are not basic Windows controls.
What is custom painted controls? and how we can test these controls?
Test it just like any other UI: Not at all. Move all code out of the callbacks into the application layer where your unit tests can execute them just like any other method.
Rationale: There is little point in testing whether "button.activate()" works; you want to know whether the your code behind the button callback works.
If you need to know whether the correct dialogs, etc., are opened, see my blog: Testing the Impossible: User Dialogs
Have the developers added support for accessibility using IAccessible? If they have, you can use active accessibility to navigate through the controls and test them that way.
If they haven't, open a bug that says that their controls can't be use by disabled people (who need a screen reader or other accessibility aid).
Once they fix that bug, you can use whatever mechanism they added to their controls to allow them to be used by screen readers and other accessibility aids to test their UI.