Can view a URL through a browser but not with CURL

Question

Hey guys,

I ran into an interesting issue today when I was working on an application I'm developing. Hopefully someone knows why this is happening / how to adjust my work flow for it!

Background:
I'm writing an application that helps students at universities get in touch with each other. The basic workflow is as follows:

1. User registers for the service
2. The application uses CURL to poll the university directory for their name
3. Store their contact info in a database table

My test site is the Rutgers University directory (http://www.acs.rutgers.edu/directory) I can access the service fine through my browser (Posts to http://www.acs.rutgers.edu/pls/pdb_p/Pdb_Display.search_results), but if I try to post the same information via CURL, I get a 404 error.

Note: I get the same error if I open that url directly from a browser and don't use their form to submit the data.

Code:
Here is the code they use on the directory site:

<fieldset>
<legend>Search For People</legend>
<div style="width:50%;margin-left:auto;margin-right:auto;">
<form method="post" action="http://www.acs.rutgers.edu/pls/pdb_p/Pdb_Display.search_results" name="thisform" onkeyup="highlight()" onclick="highlight()">
    <p>
        <label for="p_name_last">Last Name:&nbsp;&nbsp;[Required]</label><br>
        <input tabindex="1" accesskey="L" class="required" type="text" id="p_name_last" name="p_name_last" size="25" maxlength="23">
    </p> 
    <p>
        <label for="p_name_first">First Name:&nbsp;&nbsp;[or initial]</label><br> 
        <input tabindex="2" accesskey="f" type="text" id="p_name_first" name="p_name_first" size="25" maxlength="23"> 
    </p>
    <input tabindex="3" type="submit" value="Search">&nbsp;&nbsp;
</form>
</div>

And here is the code I am using to CURL the service:

<?php
$p_name_last = "doe";
$p_name_first = "";
$curlPost = 'p_name_last='  . urlencode($p_name_last) . '&p_name_first=' . urlencode($p_name_first) . '&SUBMIT=Search';
$ch = curl_init(); 
curl_setopt($ch, CURLOPT_URL, 'http://www.acs.rutgers.edu/pls/pdb_p/Pdb_Display.search_results');
curl_setopt($ch, CURLOPT_HEADER, 1); 
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); 
curl_setopt($ch, CURLOPT_POST, 1); 
curl_setopt($ch, CURLOPT_POSTFIELDS, $curlPost); 
if( ! $result = curl_exec($ch)) 
{ 
    trigger_error(curl_error($ch)); 
} 
curl_close($ch); 
echo "<pre>";
print_r($result); 
?>

Any thoughts or suggestions would be greatly appreciated!

Thanks,
Mike

Answer 1

Indeed page http://www.acs.rutgers.edu/pls/pdb_p/Pdb_Display.search_results do not exists. You have redirection to New Address: http://eas.rutgers.edu, browser can follow redirection but CURL might have a problem with it.

The web page you are trying to reach has moved!

New Address: http://eas.rutgers.edu

Please remember to update your bookmarks.

This page will automatically redirect in 10 seconds. If redirect fails, click http://eas.rutgers.edu to continue.

The web page you are trying to reach has moved!

New Address: http://eas.rutgers.edu

Please remember to update your bookmarks.

This page will automatically redirect in 10 seconds. If redirect fails, click http://eas.rutgers.edu to continue.

Answer 2

Web servers can choose what response to send after processing the request. That'll be why you are seeing the 404 - there's a resource there but it's responding in a unintuitive manner.

Anyway, I can post with curl to this page and get a response;

curl http://www.acs.rutgers.edu/pls/pdb_p/Pdb_Display.search_results --data-urlencode p_name_last=test --data-urlencode p_name_first=test

If I post with your SUBMIT=... param I get a 404. Try it with just the two name parameters.

---

Update: in fact, you can send just the last name parameter so long as the first name parameter is present - the first name parameter can be empty.