BigSERP.Com - "SERP's Up!"

SEO Blog, Tools, Glossary, Internet Marketing Consulting

  • Home
  • Tools
    • Glossary
    • Website Analysis Tools
    • Link Analysis Tools
    • Keyword Research Tools
  • Consulting
    • Internet Marketing Consulting
  • Website SEO
    • Domains
    • Site Structure
    • Conversion Design
    • Optimize URLs
    • Page Tags
    • Inbound Links
    • Reporting
    • Link Building
  • PPC/SEM
  • SEO Content
    • Website Content
    • SEO Content Writing and Installation
  • Blog Site Design
    • Website Design
  • About Us

Robots.txt Can Kill Archive.org’s Effectiveness As A Domain Name SEO Tool

June 29, 2007 by Vinny Leave a Comment

If you aren’t using Archive.org while performing your domain name SEO research, then you certainly should be… or maybe not for long? It looks like some clever chap has found a way through robots.txt to undermine the Internet Archive Wayback Machine so that the site history cannot be viewed. I won’t go into too much detail because it is a site that we are considering purchasing, however there are some interesting entries in the site’s robots.txt file:

Disallow:/cgi-bin
Disallow:/cgi-bin/link.cgi
Disallow:/cgi-bin/links.cgi
Disallow:/cgi-bin/Links.cgi
Disallow:/cgi-bin/Count.cgi
Disallow:/cgi-bin/ls1.pl
Disallow:/servlet/UserPrice
Disallow:/servlet/PrivateSale
Disallow:/servlet/buyDomDLS
Disallow:/ticker.jsp
Disallow:/searchAdvanced.jsp
Disallow:/searchPrice.jsp
Disallow:/searchPremiums.jsp
Disallow:/searchRegister.jsp
Disallow:/jsp/search_advanced.jsp
Disallow:/jsp/search_advanced_text.jsp
Disallow:/jsp/search_category.jsp
Disallow:/jsp/search_expired.jsp
Disallow:/jsp/search_keyword.jsp
Disallow:/jsp/search_letter.jsp
Disallow:/jsp/search_namefind.jsp
Disallow:/jsp/search_pricerange.jsp
Disallow:/jsp/search_similar_domains.jsp

It looks like a lot of these entries are referring to apps that crawl the web looking for expired domains, or domains for sale. At any rate, for the price this particular site owner is asking for the site in question, this use of robots.txt to prevent prospective buyers from seeing the history of a site seems a little shady. We’re still thinking about buying the domain because it’s a great name, but it’s more difficult to gauge the value of a site we don’t know the history of. And just the fact that this owner would go to such lengths to prevent people from seeing the history of the site does not reassure BigSERP’s confidence in a possible transaction.

What this means is that people parking domains, buying sites for resale, or vulturing for deleted sites now have a powerful tool in their arsenal to potentially deceive honest people looking for a good name. Who knew, it turns out robots.txt is more valuable than just a piece of code that most sites ignore!

Share this:

  • Facebook
  • LinkedIn
  • Pinterest

Related

Filed Under: SEO Tips

About Vinny

After racking up over 10 years of experience in Internet Marketing and Web Development I decided to take a division I started for a local Web Design firm and break it off into its own entity. This was June 2010 and this entity came to be known as imFORZA.

When I’m not working with my amazing team at imFORZA you will probably find me traveling, reading, playing with some new gadget or application and/or trying to help others turn their ideas into tangible, successful realities.

Leave a Reply

Your email address will not be published. Required fields are marked *

Subscribe to Blog via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Archives

  • December 2008
  • January 2008
  • August 2007
  • July 2007
  • June 2007
  • May 2007
  • April 2007

Powered by imFORZA