Robots.txt Can Kill Archive.org’s Effectiveness As A Domain Name SEO Tool

If you aren’t using Archive.org while performing your domain name SEO research, then you certainly should be… or maybe not for long? It looks like some clever chap has found a way through robots.txt to undermine the Internet Archive Wayback Machine so that the site history cannot be viewed. I won’t go into too much detail because it is a site that we are considering purchasing, however there are some interesting entries in the site’s robots.txt file:

Disallow:/cgi-bin
Disallow:/cgi-bin/link.cgi
Disallow:/cgi-bin/links.cgi
Disallow:/cgi-bin/Links.cgi
Disallow:/cgi-bin/Count.cgi
Disallow:/cgi-bin/ls1.pl
Disallow:/servlet/UserPrice
Disallow:/servlet/PrivateSale
Disallow:/servlet/buyDomDLS
Disallow:/ticker.jsp
Disallow:/searchAdvanced.jsp
Disallow:/searchPrice.jsp
Disallow:/searchPremiums.jsp
Disallow:/searchRegister.jsp
Disallow:/jsp/search_advanced.jsp
Disallow:/jsp/search_advanced_text.jsp
Disallow:/jsp/search_category.jsp
Disallow:/jsp/search_expired.jsp
Disallow:/jsp/search_keyword.jsp
Disallow:/jsp/search_letter.jsp
Disallow:/jsp/search_namefind.jsp
Disallow:/jsp/search_pricerange.jsp
Disallow:/jsp/search_similar_domains.jsp

It looks like a lot of these entries are referring to apps that crawl the web looking for expired domains, or domains for sale. At any rate, for the price this particular site owner is asking for the site in question, this use of robots.txt to prevent prospective buyers from seeing the history of a site seems a little shady. We’re still thinking about buying the domain because it’s a great name, but it’s more difficult to gauge the value of a site we don’t know the history of. And just the fact that this owner would go to such lengths to prevent people from seeing the history of the site does not reassure BigSERP’s confidence in a possible transaction.

What this means is that people parking domains, buying sites for resale, or vulturing for deleted sites now have a powerful tool in their arsenal to potentially deceive honest people looking for a good name. Who knew, it turns out robots.txt is more valuable than just a piece of code that most sites ignore!