Frame 609

When is a “404: File Not Found” not really a “404″?  If you’ve set up a custom 404 page in IIS, it very well might be… err not be.

I don’t think anyone is a stanger to having clicked a bum link somewhere, and ended up with ”Error 404: File Not Found” glaring at them.  The message seems pretty self explanatory, but unless you’ve done a little reading about how HTTP works, you aren’t going to know that the “404″ is a actually a message from your web server to the user’s browser (or whatever other user agent has requested the missing page)… it’s not just a code for an admin to reference when tracking down an error.

Setting up a custom 404 page is a pretty common task these days.  Web Developers and Web Designers want to display a page that looks like the rest of their site and that has a friendlier message for their users than the canned one the web server generates.  Those with SEO in mind are also likely to present a sitemap on the customized 404 page (also a good usability practice).  Some out there with more of a back-end/admin mentally might even set up some custom logging or email generation when a 404 page is served to let them know someone is trying to hit something that either is no longer there or never existed.

But my guess is that most developers that are using either Classic .ASP or even ASP.Net with IIS, think that making that simple change in pathing to their new prettied up/logging page (either in IIS directly for .ASP or in web.config for ASP.Net) is all they need to do to have set up the custom 404.  I remember back when I was first learning how to do it from sites like 4GuysfromRolla (when ASP was in it’s heyday), all you were really told to do after creating your nice looking HTML page, was to change the pathing to it in IIS… that was it. 

But it’s not.  Yes, making the change in the IIS properties will serve up your custom prettied-up page when someone requests a file that doesn’t exist, but there’s a problem…  Your server is still sending the browser or other user agent a “200 OK” status message.

Don’t believe me?  Grab the Web Developer Toolbar for Firefox, try and hit a bad URL on your site (assuming of course you’re running IIS) and look at the headers, by going to “Information->View Response Headers”.  You should see “200 OK” as the last line.   Well gee.. it served up that error page OK…   But the thing is, it didn’t serve up the page you requested.  What it should have sent the user agent was a 404 error, letting that agent know that the requested page doesn’t exist on the server.  This even happens in ASP.Net 2.0 when you use the built-in custom error page mechanism in Web.config… the page you define has to explicity send a 404 Status, otherwise IIS will just send a HTTP Status message of “200 OK”.

Why is it a problem?

Well, it’s really a matter of building things the way we’re supposed to be building them. It’s about following the standards set up by the W3C for how the web is supposed to work. We all know that the browser makers haven’t always supported the standards that were set in place by the W3C… and we all know how much complaining we did because of it. So this is one area where it’s up to us to try and adhere to the standard.  It makes for a better web.

There’s also an SEO benefit to this. When a Search Engine’s crawler comes by your site, the 404’s should tell it that it can drop whatever link it’s trying to index. Of course there are other types of redirects that might be more appropriate. But I’ll leave it to dedicated SEOs to give their reasons.

So what’s the solution?

You need to have your custom page explicitly change the HTTP headers to send a 404 status message.  There are a ton of articles around the web that talk about how to do this, but here’s a quick rundown.  You’ll use the Status property of Response object to do this in either environment (ASP or .Net).   And in .Net you’ll specify you’re custom error page in your web.config file.

Obviously you’ll need to make syntax alterations for your environment, but it’s basically identical between Classic ASP and ASP.Net:

Response.Status=”404 Not Found”   

Yup… that simple.  It just angers me that IIS doesn’t automatically send the 404 status when you set up your custom page.  It should at least give you some indication that it isn’t going to do so… or give you the option to automatically returned when you are setting the pathing for the custom page.   I can see some reasons why you might not want to have it sent all the time — for instance when running a campaign that provides “custom urls” to a list of recipients, you don’t really want to have to create a few thousand folders.  But really, when you are making the change in IIS, you aren’t really making the change unless you do it explicitly.  

 One last thing…

In the course of double checking my info, and finding the appropriate links for the post, I came across that in reality a 410 (Gone) status is what should be served.  I think I’ll have to do some more looking into that and see how many people actually use them.