How can I use Java to read/draw/interact with another Java window/applet

Question

I am interested in writing a Java program which does the following.

• Attach to a currently running Swing window or applet, running in another JVM
  • Alternatively, load a Java application so as to be able to do the above
• Read colors from the window
• Send mouse and keyboard events to the window

I am hoping to make some sort of testing and automation tool that interacts with GUIs directly.

I am looking for some advice on how to accomplish any and all of the steps above. Thanks in advance.

Answer 1

If you have control over the 'tested' application, I think your best bet might be to create a server which allowed the tester program to proxy calls to an internal java.awt.Robot.

Answer 2

It's not going to be so easy to capture the output of a another process - it will be simplest if your test app launches directly the swing app, in the same java VM.

You can then call paint(Graphics g) on the JFrame, passing the component an off screen graphics (BufferedImage - details here.) You can send input events to the AWT event queue via EventQueue.postEvent(AWTEvent) - this can be used to simulate AWT input.

However, have you surveyed the existing test frameworks out there? FEST has a test framework specifically for manipulating and verifying UI. There is also abbot, older by favored by some. There are also test frameworks that concentrate on function and state rather than screen grabs and input events. These are not better/worse, but complementary. State UI testing includes SwingUnit, and UISpec4J.

There are a lot of good frameworks out there, so it might pay to do a little research before building yet another one!

EDIT: To launch an application, instead of running

java -cp ... a.b.c.AppToTest

You run

java -cp ... TestWrapper AppToTest

and implement TestWrapper like this

public class TestWrapper {
/* args[0] - class to launch. Remainder of args passed to launched app.*/
   public static void main(String[] args) throws Exception
   {
      Class app = Class.forName(args[0]);
      Method main = app.getDeclaredMethod("main", new Class[] { (new String[1]).getClass()});
      String[] appArgs = new String[args.length-1];
      System.arraycopy(args, 1, appArgs, 0, appArgs.length);
      main.invoke(null, appArgs);
      // now you have just launched another application inside the same VM
   }

}

Once you have launched the app, you can wait for the app to start, and poll Window.getOwnerlessWindows to find any top-level windows the app creates.

A more direct approach is to install your own RepainManager - you canjust delegate to the existing one. This is invoked for all window drawing operations, an so you get right in the heart of the window hierarchy,

You can also register to listen to all events on the AWT EventQueue. That will also give you an inside view on what is happening in the app, and from that you can determine which windows are created, in focus etc..

Answer 3

You'll need to write and support an specific protocol ( of your own invention ) in the target app.

You can do it with RMI, Sockets or some other form of RCP ( Protobuf, JSON, WebService etc. )

The point is, you have to open a port in the app and be able to understand instructions from an external source.

If what you're looking a a way to test it, I've heard Abbot is a good option. Take a look.

If you want to manipulate an application ( of which you don't have the source ) that may be more complicated.