Does TestNG guarantee @BeforeSuite methods are executed before @BeforeTest methods?

Question

BACKGROUND: My goal is to code a TestNG-Selenium system that runs self-contained (no strings to maven or ant plugins; just java). It must allow for test cases to accept parameters including the browser and the domain url. When the TestRunner instantiates these test cases, the browser and domain are used to get a Selenium object to perform it's testing.

PROBLEM: Only one test class per suite succeeds in getting the domain parameter (in a @BeforeSuite method) before attempting to get a Selenium object (in a @BeforeTest). The test classes that do not receive the domain have a null selenium object b/c it cant be instantiated.

CODE: The XmlClasses are each contained within their own XmlTest and all three are in a single XmlSuite. The suite contains the in the order of TestClass1, TestClass2, then TestClass3. The test classes themselves are subclasses of 2 layers of abstract base classes that includes functionality to initialize injected variables and subsequently get an instance of Selenium. The purpose for this is to test one or multiple applications (on multiple domains) with as little repeated code as possible (ie: Selenium instantiation is in the root base class because it's common to all tests). See the methods below for details.

// Top-most custom base class
abstract public class WebAppTestBase extends SeleneseTestBase
{
        private static Logger logger = Logger.getLogger(WebAppTestBase.class);
        protected static Selenium selenium = null;
        protected String domain = null;
        protected String browser = null;

        @BeforeTest(alwaysRun = true)
        @Parameters({ "selenium.browser" })
        public void setupTest(String browser)
        {
                this.browser = browser;
                logger.debug(this.getClass().getName()
                                + " acquiring Selenium instance ('" + this.browser + " : " + domain + "').");
                selenium = new DefaultSelenium("localhost", 4444, browser, domain);
                selenium.start();
        }

}

// Second level base class.
public abstract class App1TestBase extends WebAppTestBase
{

        @BeforeSuite(alwaysRun = true)
        @Parameters({"app1.domain" })
        public void setupSelenium(String domain)
        {
                // This should execute for each test case prior to instantiating any Selenium objects in @BeforeTest
                logger.debug(this.getClass().getName() + " starting selenium on domain '" + domain+ "'.");
                this.domain = domain;
        }
}

// Leaf level test class
public class TestClass1 extends App1TestBase
{
        @Test
        public void validateFunctionality() throws Exception
        {
                // Code for tests go here...
        }
}

// Leaf level test class
public class TestClass2 extends App1TestBase
{
        @Test
        public void validateFunctionality() throws Exception
        {
                selenium.isElementPresent( ...
                // Rest of code for tests go here...
                // ....
        }
}


// Leaf level test class
public class TestClass3 extends App1TestBase
{
        @Test
        public void validateFunctionality() throws Exception
        {
                // Code for tests go here...
        }
}

OUTPUT: TestCase3 runs correctly. TestCase1 and TestCase2 fails. Stack trace gets generated...

 10:08:23 [DEBUG RunTestCommand.java:63] - Running Tests.
 10:08:23 [Parser] Running:
  Command line suite
  Command line suite

[DEBUG App1TestBase.java:49] - TestClass3 starting selenium on domain 'http://localhost:8080'.
 10:08:24 [DEBUG WebAppTestBase.java:46] - TestClass2 acquiring Selenium instance ('*firefox : null').
 10:08:24 [ERROR SeleniumCoreCommand.java:40] - Exception running 'isElementPresent 'command on session null
 10:08:24 java.lang.NullPointerException: sessionId should not be null; has this session been started yet?
        at org.openqa.selenium.server.FrameGroupCommandQueueSet.getQueueSet(FrameGroupCommandQueueSet.java:216)
        at org.openqa.selenium.server.commands.SeleniumCoreCommand.execute(SeleniumCoreCommand.java:34)
        at org.openqa.selenium.server.SeleniumDriverResourceHandler.doCommand(SeleniumDriverResourceHandler.java:562)
        at org.openqa.selenium.server.SeleniumDriverResourceHandler.handleCommandRequest(SeleniumDriverResourceHandler.java:370)
        at org.openqa.selenium.server.SeleniumDriverResourceHandler.handle(SeleniumDriverResourceHandler.java:129)
        at org.openqa.jetty.http.HttpContext.handle(HttpContext.java:1530)
        at org.openqa.jetty.http.HttpContext.handle(HttpContext.java:1482)
        at org.openqa.jetty.http.HttpServer.service(HttpServer.java:909)
        at org.openqa.jetty.http.HttpConnection.service(HttpConnection.java:820)
        at org.openqa.jetty.http.HttpConnection.handleNext(HttpConnection.java:986)
        at org.openqa.jetty.http.HttpConnection.handle(HttpConnection.java:837)
        at org.openqa.jetty.http.SocketListener.handleConnection(SocketListener.java:245)
        at org.openqa.jetty.util.ThreadedServer.handle(ThreadedServer.java:357)
        at org.openqa.jetty.util.ThreadPool$PoolThread.run(ThreadPool.java:534)

I appreciate any information you may have on this issue.

Answer 1

I think the problem is that your @BeforeSuite method is assigning a value to a field, but you have three different instances, so the other two never get initialized.

Remember that @BeforeSuite is only run once, regardless of what class it belongs to. As such, @Before/AfterSuite methods are usually defined on classes that are outside the entire test environment. These methods should really be static but I decided not to enforce this requirement because it's sometimes impractical.

I think a better way to approach your problem is to look at your domain field as an injected resource that each of your test will receive from Guice or other dependency injection framework.