TestNG - Factories and Dataproviders

Question

Background Story

I'm working at a software firm developing a test automation framework to replace our old spaghetti tangled system.

Since our system requires a login for almost everything we do, I decided it would be best to use @BeforeMethod, @DataProvider, and @Factory to setup my tests. However, I've run into some issues.

Sample Test Case

Lets say the software system is a baseball team roster. We want to test to make sure a user can search for a team member by name.

(Note: I'm aware that BeforeMethods don't run in any given order -- assume that's been taken care of for now.)

@BeforeMethod
public void setupSelenium() {
    // login with username & password
    // acknowledge announcements
    // navigate to search page
}

@Test(dataProvider="players")
public void testSearch(String playerName, String searchTerm) {
    // search for "searchTerm"
    // browse through results
        // pass if we find playerName
    // fail (Didn't find the player)
}

This test case assumes the following:

• The user has already logged on (in a BeforeMethod, most likely)
• The user has already navigated to the search page (trivial, before method)
• The parameters to the test are associated with the aforementioned login

The Problems

So lets try and figure out how to handle the parameters for the test case.

Idea #1

This method allows us to associate dataproviders with usernames, and lets us use multiple users for any specific test case!

@Test(dataProvider="players")
public void testSearch(String user, String pass, String name, String search) {
    // login with user/pass
    // acknowledge announcements
    // navigate to search page
    // ...
}

...but there's lots of repetition, as we have to make EVERY function accept two extra parameters. Not to mention, we're also testing the acknowledge announcements feature, which we don't actually want to test.

Idea #2

So lets use the factory to initialize things properly!

class BaseTestCase {

    public BaseTestCase(String user, String password, Object[][] data);

}

class SomeTest {

    @Factory
    public void ...

}

With this, we end up having to write one factory per test case... Although, it does let us have multiple users per test-case.

Conclusion

I'm about fresh out of ideas. There was another idea I had where I was loading data from an XML file, and then calling the methods from a program... but its getting silly.

Any ideas?

Answer 1

First, it seems like you are trying to do too much in each test case. For instance, if you are searching for something, why should you need to test navigation as part of searching?

Second, it seems like your requirements are unclear. Can you create a test where you send it a single search term and get back a single result? Seems like this should be your first test.

Third, why can't an authorized session connection/object be a requirement for a test? From the looks of your code, it looks like you are making some kind of call via HTTP. Even if you aren't, it looks like you must be using some kind of broker to send a message to your app, given that you are having to pass along user name and pass on each request... Why not just automate that whole thing into an "authorized broker" object that gives you a way to send your request along a pre-authorized connection?

Then just pass the authorized broker into your tests to handle your message passing. Or better yet, set up your authorized broker in your pre-test (BeforeMethods?) functions, and make it available as a class member variable.

Answer 2

Zak, thanks for your reply! A few responses.

Doing too much You're telling me! :P Our application is archaic, the only way to test things is testing them all together.

Unclear Requirements My problem isn't developing test cases per-se, its more setting up the test-case environment. (Although that's a good point, I'll put that into our testing documents!)

Brokers That's the point of PageObjects in Selenium... My problem though is configuring them, not using them. To boil it down, how to do point three with TestNG and Selenium in a non-kludgey way?

Answer 3

I'm still not sure I understand your problem...

Are you not satisfied with your second attempt, i.e. having to write a @Factory per test case? Then why not put the @Factory annotation on the base class constructor? This way, all your subclasses need to do is call super and you're done.

What are you trying to achieve exactly?

Answer 4

Cederic,

I should have made an account from day one, I can't actually edit my old post anymore, or reply to comments (at this point). Let me try and expand things a little.

Here's an example of what I've come up with so far. I guess I answered my own question... this works, but its a little nasty.

My question is now, I have all my tests in one factory. That's no good for several reasons. One, I have to add them all by hand (or put them in reflectively). Two, I have to run everything essentially as one suite. Any ideas?

package example2;

import java.lang.reflect.Method;
import java.util.HashMap;
import org.testng.annotations.*;

public class WebTestBase {

  protected String host, username, password;
  protected HashMap<String, Object[][]> dataSet;

  public WebTestBase(String host, String username, String password, HashMap<String, Object[][]> dataSet) {
    this.host = host;
    this.username = username;
    this.password = password;
    this.dataSet = dataSet;
  }

  @DataProvider(name="dataSet")
  public Object[][] dataSet(Method m) {
    return dataSet.get(m.getName());
  }

  @BeforeMethod
  public void login() {
    System.out.println("Logging in to " + host + " with " + username + ":" + password);
  }

  @AfterMethod
  public void logout() {
    System.out.println("Logging out!");
  }
}

---

package example2;

import java.util.HashMap;

import org.testng.annotations.Factory;

public class WebTestFactory {
  @Factory
  public Object[] factory() {
    HashMap<String, Object[][]> dataSetOne = new HashMap<String, Object[][]>();
    dataSetOne.put("searchRoster", new Object[][] {
          {"mcguire", "McGuire, Mark"},
          {"ruth", "Ruth, Babe"}
        });

    HashMap<String, Object[][]> dataSetTwo = new HashMap<String, Object[][]>();
    dataSetTwo.put("addPlayer", new Object[][] {
          {"Sammy Sosa", 0.273}
        });

    Object[] tests = new Object[] { 
        new SearchTest("localhost", "user", "pass", dataSetOne),
        new AddTest("localhost", "user", "pass", dataSetTwo)
      };

    return tests;
  }
}

---

package example2;

import java.util.HashMap;
import org.testng.annotations.Test;

public class SearchTest extends WebTestBase {

  public SearchTest(String host, String username, String password,
      HashMap<String, Object[][]> dataSet) {
    super(host, username, password, dataSet);
  }

  @Test(dataProvider="dataSet")
  public void searchRoster(String searchTerm, String playerName) {
    System.out.println("Searching for " + searchTerm);
    System.out.println("I found " + playerName + " in the search results!");
  }

}

---

package example2;

import java.util.HashMap;

import org.testng.annotations.Test;

public class AddTest extends WebTestBase {

  public AddTest(String host, String username, String password,
      HashMap<String, Object[][]> dataSet) {
    super(host, username, password, dataSet);
  }

  @Test(dataProvider="dataSet")
  public void addPlayer(String playerName, double battingAvg) {
    System.out.println("Adding " + playerName + " with avg " + battingAvg);
  }
}