"Session expired" and "Please enable cookies" warnings when POSTing website form through cURL

Question

I have been developing a program that is able to login to AWeber.com, and perform mass imports of data. The script uses the PHP cURL libraries, along with their CookieJar settings to spoof a normal user with a browser.

The script works perfectly, allowing login, and the changing of lists, but when it comes to posting the form data (in the submitData function) the script invariably fails. Every time the website will output a web page indicating that the session has expired, and asking the "user" to login again. The page also requests the "user" to enable cookies in their browser.

I have spend the last couple of days diagnosing the problem, and it has me completely stumped. The CURLOPT_VERBOSE setting indicates that cURL is passing the cookies to the website, the cookiejar file contains the cookies and additional factors such as Referer and Javascript requirements have been eliminated from the cause factors.

I would appreciate any suggestions as to why this may be occurring and solutions to the problem. Class and code that causes error is displayed below. Code is marked where error occurs.

<?php
class AWeber {
  private $cj;

  public function __construct() {
    $this->cj = tempnam('/tmp', 'mlcookies_');
  }

  private function postQuery( $url, $array ) {
    $cu = curl_init();
    curl_setopt( $cu, CURLOPT_URL, $url );
    curl_setopt( $cu, CURLOPT_POST, true );
    curl_setopt( $cu, CURLOPT_POSTFIELDS, $array );
    curl_setopt( $cu, CURLOPT_COOKIEJAR, $this->cj );
    curl_setopt( $cu, CURLOPT_COOKIEFILE, $this->cj );
    curl_setopt( $cu, CURLOPT_RETURNTRANSFER, true );
    curl_setopt( $cu, CURLOPT_FOLLOWLOCATION, true );
    curl_setopt( $cu, CURLOPT_HEADER, true );
    curl_setopt( $cu, CURLOPT_USERAGENT, 'Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0)' );

    $return = curl_exec($cu);
    echo $return;
    curl_close($cu);
    return $return;
  }

  private function getQuery( $url ) {
    $cu = curl_init();
    curl_setopt( $cu, CURLOPT_COOKIEJAR, $this->cj );
    curl_setopt( $cu, CURLOPT_COOKIEFILE, $this->cj );
    curl_setopt( $cu, CURLOPT_URL, $url );
    curl_setopt( $cu, CURLOPT_RETURNTRANSFER, true );
    curl_setopt( $cu, CURLOPT_FOLLOWLOCATION, true );
    curl_setopt( $cu, CURLOPT_HEADER, true );
    curl_setopt( $cu, CURLOPT_USERAGENT, 'Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0)' );

    $return = curl_exec($cu);
    curl_close($cu);
    echo $return;
    return $return;
  }

  public function login( $user, $pass ) {
    $this->getQuery( "https://www.aweber.com/login.htm" ); // Get page cookie checks
    $query = array(
      '_method' => 'POST',
      'data[Account][username]' => $user,
      'data[Account][password]' => $pass,
      'data[Account][remember_login]' => '1'
    );
    $return = $this->postQuery( "https://www.aweber.com/login.htm", $query );
    $this->getQuery( "https://www.aweber.com/users/" );
    if ( !$return ) return false;  
    if ( strpos($return, '<div class="aw-status-headline">Error</div>') === false ) {
      return true;
    } else {
      return false;
    }
  }

  public function setList( $list ) {
    $return = $this->getQuery( "https://www.aweber.com/users/lists/change/" . $list );
    if ( !$return ) return false;
    if ( strpos($return, '<option selected="selected" id="listSelectionActiveOption" value="' . $list . '">' ) === false ) {
      return false;
    } else {
      return true;
    }
  }

  public function submitData( $text, $note ) {
    $query = array(   
      'upload_file' => '1',
      'data[ImportWizard][leads]' => $text,
      'data[ImportWizard][delimiter]' => 'TAB',
      'data[ImportWizard][customer_note]' => $note,
      'data[ImportWizard][use_automation]' => '1',
      'cmd' => 'Next',
    );
    $return = $this->postQuery( "https://www.aweber.com/users/lead_imports", $query );
    echo $return;
    if ( !$return || strpos($return, '<h1>Step 2</h1>') === false ) return false;

    $query = array(
      'columnArray' => '',
      'columnArray' => '',
      'data[ImportWizard][column0]' => 'name',
      'data[ImportWizard][column1]' => 'email',
      'cmd' => 'Save',
      );
    $return = $this->postQuery( "https://www.aweber.com/users/lead_imports", $query );
    if ( !$return || strpos($return, '<div class="aw-status-headline">Import Queued</div>') === false ) return false;
    return true;
  }

}

$aw = new AWeber(); // Create new AWeber interface class instance
$aw->login( $aUser, $aPass ) or trigger_error('Login failed', E_USER_ERROR); // Login
$aw->setList( 'list1' ) or trigger_error('List change 1 failed', E_USER_ERROR); // Change mailing list to 'list1'

// *** CODE WILL FAIL HERE WITH "Data submit 1 failed" ERROR ***
$aw->submitData( "Test\tTesterrr\nTest2\tTesterrr2\nTest3\tTesterrr3\n", "Testing Testing Testing Testing Testing Testing Testing" ) or trigger_error('Data submit 1 failed', E_USER_ERROR); // Submit data
$aw->setList( 'list2' ) or trigger_error('List change 2 failed', E_USER_ERROR); // Change mailing list to 'list2'
?>

Answer 1

It may be because you are closing the curl handle with the curl_close in between calls during the session. It should only be closed after the session usage is completed.

Answer 2

You may need to read the response header data that is sent back from the first query. Then pass the session information (JSSESSION/PHPSESSID/etc) back as cookie information.

I think your post data script is resetting the cookie data (not sure how). Use Firebug to monitor the headers that are sent and received when you perform these operations. Then check the cookie files to ensure they are not being overridden.

Answer 3

You should check the cookies file. If everything is OK, it should have information. If not, perhaps you should use cwd() to specify an absolute path.

Answer 4

I think Turnkey is correct.

It may be because you are closing the curl handle with the curl_close in between calls during the session. It should only be closed after the session usage is completed.

You must not open two different curl session. Try doing everything in one session then close the session.

Try adding curl_close() in class's destructor. Store curl session in a variable. And access it using $this->curl_session. Here is an example

<?php
    class AWeber {
    $curl_session;
    $cj;
    function __construct() {
        $this->curl_session = curl_init();
    }
    function __destruct() {
        if($this->curl_session) {
            curl_close($this->curl_session);
        }
    }
    function doWhatever() {
            curl_setopt( $this->curl_session, CURLOPT_COOKIEJAR, $this->cj );
            curl_setopt( $this->curl_session, CURLOPT_COOKIEFILE, $this->cj );
            curl_setopt( $this->curl_session, CURLOPT_URL, $url );
            curl_setopt( $this->curl_session, CURLOPT_RETURNTRANSFER, true );
            curl_setopt( $this->curl_session, CURLOPT_FOLLOWLOCATION, true );
            curl_setopt( $this->curl_session, CURLOPT_HEADER, true );
            curl_setopt( $this->curl_session, CURLOPT_USERAGENT, 'Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0)' );

            $return = curl_exec($this->curl_session);

    }
}
?>