Mod Rewrite Producing 404 When Crawled (works fine when viewed in browser)

Question

I have the following code in my .htaccess:

RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]

My pretty-link converting is done on my index.php. If the user is looking for something that doesn't exist, a 404 Header is produced, otherwise content is show.

That is all fine and dandy, however when a crawler or something tries to view the HTML, they are getting a 404 error.

For example:

http://www.jasonleodurbin.com/portfolio That link should work fine.

If you try to validate it at an HTML validator, it doesn't work. The crawler says it is getting a 404. I am getting the same for Facebook share.

I've tried removing the 404 error header, and I am still getting the same problem.

What is the deal? Any suggestions?

EDIT:

New .htaccess:

RewriteEngine On
RewriteCond %{HTTP_HOST} ^jasonleodurbin\.com$ [NC]
RewriteRule ^(.*)$ http://www.jasonleodurbin.com/$1 [R=301,L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^([a-z]+)/$ index.php?go=$1&app=1 [NC,L,QSA]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^([a-z]+)?success=true index.php?go=$1&success=1 [NC,L,QSA]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^([a-z]+) index.php?go=$1 [NC,L,QSA]

Processing:

globals.php

if(!isset($_GET['go']) || $_GET['go'] == "") $_GET['go'] = "home";

header.php

if(isset($_GET['go']) && !@fopen(strtolower($_GET['go']).".php",'r')){
    //header("HTTP/1.0 404 Not Found");
    define("FAIL",true);
    $_GET['go'] = "error";
}
else {
    define("FAIL",false);
Header('HTTP/1.1 200');
}
if(FAIL) define("GOSUB",DIR);
if(isset($_GET['app'])) define("GOSUB","../");
else define("GOSUB","");

I've also tried this for the index.php:

<?Header('HTTP/1.1 200 OK');?>
<? require_once("header.php");?>
<? require_once(strtolower($_GET['go'].".php"));?>

    </div>
</div>
<?
    if(!defined("FOOTER"))define('WP_USE_THEMES', false);
    define("FOOTER",true);
    include("blog/index.php");
?>
<!--[if IE]>
<a href='http://www.google.com/chrome' title='Get Google Chrome : A Better Way To Browse' class='noie sprite-1'>IE Sucks</a>
<![endif]-->
<!--[if lt IE 7]>
<div style='position:absolute;top:60px;left:0px;'><b>IE 6</b>?! Dude, upgrade. <br>Click the link above to get Google Chrome.</div>
<![endif]-->
</body>
</html>

Something is sending the 404 before I can even send 200.

Answer 1

I do get the portfolio page show up, but with a 404 header. So, check the code that sets the 404 header, the error is somewhere in there.

Seems to me that you are using fopen only to see if the file exists (if it fails opening, it doesn't exist). Use file_exists() for that: http://php.net/file_exists

Answer 2

You are seemingly using Wordpress. Wordpress sets any page that it does not find as 404.

Also you may be interested in plugin Link Juice Keeper. It redirects all 404 pages to front page using 301 redirect. And here are Wordpress 404 tips and tricks.

Answer 3

Are you sure there's no other place in code where another 404 maybe sent? if no other one find, then it seems to be an issue with your configuration, try sending an opposite 200 OK header when the page is approved.

BTW don't do fopen(strtolower($_GET['go']), the user may set $_GET['go'] to http://his.domain.com/page, and you end up including a page from his server if PHP's configurations allow that (allow_url_fopen).