Do I need to send a 404?

Question

We're in the middle of writing a lot of URL rewrite code that would basically take ourdomain.com/SomeTag and some something dynamic to figure out what to display.

Now if the Tag doesn't exist in our system, we're gonna display some information helping them finding what they were looking for.

And now the question came up, do we need to send a 404 header? Should we? Are there any reasons to do it or not to do it?

Thanks

Nathan

Answer 1

You aren't required to, but it can be useful for automated checkers to detect the response code instead of having to parse the page.

Answer 2

I say do it - if the user is actually an application acting on behalf of the user (i.e. cURL, wget, something custom, etc...) then a 404 would actually help quite a bit.

Answer 3

You have to keep in mind that the result code you return is not for the user; for the standard user, error codes are meaningless so don't display this info to the user.

However think about what could happen if the crawlers access your pages and consider them valid (with a 200 response); they will start indexing the content and your page will be added to the index. If you tell the search engine to index the same content for all your not found pages, it will certainly affect your ranking and if one page appears in the top search results, you will look like a fool.

Answer 4

I certainly send proper response codes in my applications, especially when I have database errors or other fatal errors. Then the search engine knows to give up and retry in 5 mins instead of indexing the page. e.g. code 503 for "Service Unavailable" and I also send a Retry-After: 600 to tell it to try again...search engines won't take this badly.

404 codes are sent when the page should not be indexed or doesn't exist (e.g. non-existent tag)

So yes, do send status codes.