HTTP Response Status Code 304

If you have a dynamic website that is search engine friendly, chances are your entire database of some entity such as a list of products is made available that can be easily reached by crawling by the bots. Doing this comes with a cost. The bots keep crawling regularly, even at a slower rate, and you have to render those dynamic pages one after the other costing you both bandwidth and CPU cycles. To avoid this, you can write your dynamic pages such that they are “last modified” aware. That is, say you have a product page that lists the product details as well as any comments by the user. While most of the product details itself is relatively static, content such as comments could be changing every few days especially so for a product that has launched in the first few months. Ofcourse, if you do dynamic pricing, then that’s different. But even then, people seldom search based on the exact price string (people want to know if a product is available below a certain price and not an exact price, so don’t bother to worry that your latest price is not indexed. Besides, by the time it’s indexed to the time it’s available in the search results, your price may have changed yet again).

So, how do you make your dynamic page “last modified” aware? The way it works is, bots use a special http request which is a conditional http GET request that passes a special if-modified-since header with a specific date. So, the bot is essentially asking you to respond with the full content only if the content has modified since a given date. Otherwise, you can just respond with a status code of 304 which tells the bot that there is no change. So, search engine crawlers like that of Google which maintains the last time they have crawled uses these conditional requests so that they can avoid the same bandwidth and cpu cycles as you.

How do you know if you are making use of this functionality? It’s easy. Check in your log files and see if you have any 304 response codes against Google bot or other search engine bot requests. If you always see 200 and never 304, then you are not using this feature.

Leave a comment

Filed under Search Indexing, Tech - Tips

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s