Google to Introduce New Meta Tags

The ‘unavailable_after’ meta tag will soon be recognized by Google according to Dan Crow, Director of Crawl Systems at Google and Jill Whalen of High Rankings. Google is coming out with a new tag called “unavailable_after” which will allow people to tell Google when a particular page will no longer be available for crawling. So, it gives you the option to tell Google when Googlebot should no longer crawl that page.

This tag comes in handy when you have a limited time offer promotional page, and on this page, the promotion will expire on a specific date. For instance, if you have a special offer on your site that expires on a particular date, you might want to use the unavailable_after tag to let Google know when to stop indexing it. Or perhaps you write articles that are free for a particular amount of time, but then get moved to a paid-subscription area of your site. By using the "unavailable_after" tag, you can tell Google that they should not crawl this page, after the promotion expires. There are several practical scenarios for this tag.

For example, if you have a page that you would like to be removed from the search results at 6pm EST on July 23, 2007, you would add the following Meta tag:

meta content="unavailable_after: 23-Jul-2007 18:00:00 EST" name="GOOGLEBOT"

Once Google crawls the page and sees this Meta tag, it will take about a day for the page to be removed from the search results. Also this tag is only supported with web search. To remove the page completely from Google, including the cache copies, use the Google removal tool.
Google has also added support to control access to other types of documents, such as PDF files, audio, xls documents and so on via new X-Robots-Tag directive in the HTTP Header used to serve the file.

Here are examples of how to use the X-Robots-Tag:

Don't index this document:

X-Robots-Tag: noindex

Don't show cache or snippet in search results for this file:

X-Robots-Tag: noarchive, nosnippet

Don't index after a specific date (i.e. the "unavailable_after" tag):

X-Robots-Tag: unavailable_after: 7 Jul 2007 16:30:00 GMT

You can also combine these tags as well.