BigSERP.Com - "SERP's Up!"

SEO Blog, Tools, Glossary, Internet Marketing Consulting

  • Home
  • Tools
    • Glossary
    • Website Analysis Tools
    • Link Analysis Tools
    • Keyword Research Tools
  • Consulting
    • Internet Marketing Consulting
  • Website SEO
    • Domains
    • Site Structure
    • Conversion Design
    • Optimize URLs
    • Page Tags
    • Inbound Links
    • Reporting
    • Link Building
  • PPC/SEM
  • SEO Content
    • Website Content
    • SEO Content Writing and Installation
  • Blog Site Design
    • Website Design
  • About Us

Using Robots.txt To Disallow Domains From Spidering Your Site

July 13, 2007 by Vinny Leave a Comment

As SEO practitioners, there’s often a good reason to exclude a certain search engine or site spider from looking at your pages. For example, some people who do not want their site archived over time by Archive.org will take action to block them from doing so. That’s where the robots.txt kicks in; set one up on your server, and most websites will adhere to your commands as far as what they are allowed to spider on your site. If you wish to prevent a specific site spider from indexing your website’s content, then configure your robots.txt file this way:

User-agent: Name_of_Robot
Disallow: /

Here are some of the most common site robot names that you might want to exclude:

Archive.org: ia_archiver
Google: googlebot
Yahoo: Slurp

 Regardless of what you’re trying to do with robots.txt, the most important thing is to make sure you’re doing it right. A small typo in your config file can cause disastrous results. So make sure to use the following resources to set yours up properly.

For thorough lists of robots files you can exclude, check these sites:

http://www.hostsun.com/gr/bots_index.php

http://www.robotstxt.org/wc/active.html

For thorough documentation as to how robots.txt files work, and how to properly set them up, check out these sites: 

http://www.robotstxt.org/wc/faq.html

http://en.wikipedia.org/wiki/Robots.txt 

Share this:

  • Facebook
  • LinkedIn
  • Pinterest

Related

Filed Under: SEO Tips

About Vinny

After racking up over 10 years of experience in Internet Marketing and Web Development I decided to take a division I started for a local Web Design firm and break it off into its own entity. This was June 2010 and this entity came to be known as imFORZA.

When I’m not working with my amazing team at imFORZA you will probably find me traveling, reading, playing with some new gadget or application and/or trying to help others turn their ideas into tangible, successful realities.

Leave a Reply

Your email address will not be published. Required fields are marked *

Subscribe to Blog via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Archives

  • December 2008
  • January 2008
  • August 2007
  • July 2007
  • June 2007
  • May 2007
  • April 2007

Powered by imFORZA