What Will 2009 Bring?
Wed, 31/12/08 – 22:49 | No Comment

Happy New Year from BigSERP!
We wish everyone a healthy and prosperous 2009.
Check back with us in early 2009 for a design revamp and some more great insight on the search engine marketing industry.
Thank you for …

Read the full story »
Blogging

Blogs are an important online marketing tool. Often misunderstood, those who know how to use them right will reap the rewards.

Link Building

Websites need great content, but they also must be popular. Our recommendations will increase your PageRank and other indicators of popularity.

Paid Search Marketing

Pay-per-click using Google Adwords, Yahoo Sponsored Search, Microsoft adCenter. Learn all the tricks of the trade!

SEO Tips

Search engine optimization advice. Skip paying thousands and learn the ins and outs here.

Social Marketing

Twitter, Facebook, Myspace… These are the Web 2.0 conversational marketing tools. Use them wisely.

Home » SEO Tips

Robots.txt Can Kill Archive.org’s Effectiveness As A Domain Name SEO Tool

Submitted by on Friday, 29 June 20072 Comments

If you aren’t using Archive.org while performing your domain name SEO research, then you certainly should be… or maybe not for long? It looks like some clever chap has found a way through robots.txt to undermine the Internet Archive Wayback Machine so that the site history cannot be viewed. I won’t go into too much detail because it is a site that we are considering purchasing, however there are some interesting entries in the site’s robots.txt file:

Disallow:/cgi-bin
Disallow:/cgi-bin/link.cgi
Disallow:/cgi-bin/links.cgi
Disallow:/cgi-bin/Links.cgi
Disallow:/cgi-bin/Count.cgi
Disallow:/cgi-bin/ls1.pl
Disallow:/servlet/UserPrice
Disallow:/servlet/PrivateSale
Disallow:/servlet/buyDomDLS
Disallow:/ticker.jsp
Disallow:/searchAdvanced.jsp
Disallow:/searchPrice.jsp
Disallow:/searchPremiums.jsp
Disallow:/searchRegister.jsp
Disallow:/jsp/search_advanced.jsp
Disallow:/jsp/search_advanced_text.jsp
Disallow:/jsp/search_category.jsp
Disallow:/jsp/search_expired.jsp
Disallow:/jsp/search_keyword.jsp
Disallow:/jsp/search_letter.jsp
Disallow:/jsp/search_namefind.jsp
Disallow:/jsp/search_pricerange.jsp
Disallow:/jsp/search_similar_domains.jsp

It looks like a lot of these entries are referring to apps that crawl the web looking for expired domains, or domains for sale. At any rate, for the price this particular site owner is asking for the site in question, this use of robots.txt to prevent prospective buyers from seeing the history of a site seems a little shady. We’re still thinking about buying the domain because it’s a great name, but it’s more difficult to gauge the value of a site we don’t know the history of. And just the fact that this owner would go to such lengths to prevent people from seeing the history of the site does not reassure BigSERP’s confidence in a possible transaction.

What this means is that people parking domains, buying sites for resale, or vulturing for deleted sites now have a powerful tool in their arsenal to potentially deceive honest people looking for a good name. Who knew, it turns out robots.txt is more valuable than just a piece of code that most sites ignore!

Leave a comment!

Add your comment below, or trackback from your own site. You can also subscribe to these comments via RSS.

Be nice. Keep it clean. Stay on topic. No spam.

You can use these tags:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

This is a Gravatar-enabled weblog. To get your own globally-recognized-avatar, please register at Gravatar.