Search Bots, Crawlers, And Spiders
|
|
If you are a webmaster and you review your logs, often you will see a bunch of really strange hits. They aren't humans, you can't tell their operating system or their browser! Who are these pesky little creatures who rummage around the internet all the time?
Not quite sure what I am talking about? Here is a few examples of various bots searching my website:
207.68.146.40 (msnbot.msn.com)
msnbot/1.0 (+http://search.msn.com/msnbot.htm)
This is the MSN Search bot.
207.68.146.40 (lj2070.inktomisearch.com)
Mozilla/5.0 (compatible; Yahoo! Slurp;
http://help.yahoo.com/help/us/ysearch/slurp)
This is Yahoos Search Bot.
66.249.65.147 (crawl-66-249-65-147.googlebot.com)
Mediapartners-Google/2.1
This is Googles bot, that searches your webpages for AdSense.
What is a Bot, Crawler, Spider?
These terms are all the same, they all refer to an automated program that goes from website to website caching and processing the pages for search engines. As you know, "WWW" means World Wide Web, thus "Spider" seemed like an appropriate term. Crawler is another term that just describes what it does, crawling from site to site and page to page endlessly. Bot, is actually short for "robot" and again is just an automated program to index websites.
What is the purpose of a Spider?
A spider looks at all the pages of your website, and uses that information to rank you in search engines (how high you will list in a search result), and cache a copy of your page on their server for quick reference, and if your site ever goes down. Spiders jump from link to link on the Internet and run endlessly, even if you never submit your website to a search engine, odds are your site will still be spidered.
Can I stop bots and spiders from searching my website?
Yes and no. Legitimate spiders are run by reputable organizations that follow certain rules. For instance, most companies have a policy that their robot will search for a file called "robots.txt" in the root of your website. This text file is filled with information telling the bots what and what not is allowed to be viewed. Unfortunately, there are also bad bots out there, they search the internet harvesting e-mail addresses for spam and other bad things, these bots often don't comply with the "robots.txt" standard.
How many bots are there?
It's impossible to guess how many bots are out there searching websites. On any given day I will get roughly 10 different ones check my website. Some of them only search one or two pages, others go over my entire website. Not all of them give you a good description of what they do, or who owns them. If you cut and paste their name and IP address in to Google, quite often you can find more information about what they do.
How can I get my site spidered?
As I mentioned before, if your website is up long enough, it "will" get spidered eventually. However, if you want to ensure that it gets done within a few months, go to the various search engine websites and look for the "Add URL" or "Suggest a Link" pages. DMOZ is one of the big directories which you should submit your site. When you sign up for these search engines, your website is automatically queued up to be spidered. It may take several weeks or months to actually start showing up on the search engine, even after you see the robot spidering your website.
What about pay search engines?
There are a bunch of different search engines that make you pay to have your website listed. I personally don't support these search engines, I find that most people use the big free search engines anyway. However, if you do wish to get included in some search engines faster, many have payment options which will get your site listed within a couple of days.
Ken Dennis
http://KenDennis-RSS.homeip.net/
|
|
|
Created & Maintained by Empower! CMS Web Sites
Host2Sell Web Hosting | Emarketing Workshops | Site SEO Review | FREE NewsletterHow To Boost Your Keyword Density On Your Web Site To Gain Top Positions At The Search Engines
Let's talk about what keyword density is and how to improve your keyword density on your web site. To improve your keyword density ratio there are three parts that we will need address.Part 1: ContentWhen it comes to the Search Engines content is king! Yes king! Search Engines want web sites that are rich in content. Content is words that are embedded in text on your web site. Your keyword density ratio ...(related: Search Engine Optimization)
Optimizing Your Website
Search Engine Optimization, optimizing your website for it to be visible in the search results of a search engine's query or in a search result of a directory.Basically, SEO is a marketing strategy for your site. To be included in the top ten search results of an impressive search engine/directory guarantees a return of investment. Search Engine Optimization can be viewed as a new form of publicity for a website which secures definite visitors and targeted traffic for your site.There are various Search Engines in the net today. The top rated search engine is Google. It is a prestige and go...(related: Search Engine Optimization)
Understanding Googles Algorithm
What is the Google Algorithm?An algorithm is a mathematical equation that uses certain information from your website in order to define its rankings. If you're looking for a printout of the specific equation Google uses to rank sites and how much each factor is weighed, this is not the place to get it. The actual algorithm used by Google is a secr...(related: Search Engine Optimization)
Get Personal With Google
Anyone who's been on the net any time at all knows that Google has been the search engine of choice for many for a while now. Whether that continues remains to be seen, but for now, with MSN still in development and Yahoo going through some growing pains, many still feel that Google is the leader. As such, it's worth dropping in to the Google Labs every now and then to see what's going on there.For example, Google's Personalized search offers a fun and useful way for searchers to interact with part of the mysterious and closely-guarded Google search algorithm. You can set a search profile that's stored as a cookie on your computer. The easy-to-use Google profile interface allows you to pick specific topics that interest you. Once chosen and saved, you c...(related: Search Engine Optimization)
Reciprocal Link Exchange Management Software
When shopping online for Link Management Software that can help create and automate your links directory, some important features that you should look for are as follows:The software should,
- Create the links page based on...(related: Search Engine Optimization)
Commonly Seo Terms And Jargon Used By Search Engine Gurus
What does all this SEO jargon mean.Have you ever been left scratching your head, wondering what the Search Engine Optimizers SEO's or Gurus are talking about.You are not the only one.There are so many terms used on the Internet that you almost need a degree just to understand what the experts are saying.I thought you might like to be enlightened on some of the more commonly used terms.I have divided my SEO Dictionary into two sections, Search Engine and Web site related termsThe More common Search Engine terms include,Algorithm. This is the formula used by Search Engines that determines how any web page will rank ...(related: Search Engine Optimization)
Search Bots, Crawlers, And Spiders
If you are a webmaster and you review your logs, often you will see a bunch of really strange hits. They aren't humans, you can't tell their operating system or their browser! Who are these pesky little creatures who rummage around the internet all the time?Not quite sure what I am talking about? Here is a few examples of various bots searching my website:207.68.146.40 (msnbot.msn.com)msnbot/1.0 (+http://search.msn.com/msnbot.htm)This is the MSN Search bot.207.68.146.40 (lj2070.inktomisearch.com)Mozilla/...(related: Search Engine Optimization)
Attack Smaller Searches To Get The Big Ones!
Searching online can not only be fun, but you sometimes need to be downright inventive.For those of you who have been searching online for years now, you've probably adapted ways & techniques to find what you are looking for quickly.Unfortunately when it comes to promoting your site, nothing out there is classified as a quick fix. If people say it's a quick fix, it probably has a major risk attached to it.Your ultimate goal when promoting your site should be to acquire some of those "Big" search terms online! For example, "Marketing Tips" or "Marketing" in general. The only way to ever come close to the Big Dogs is to start small. Increase the search term and decrease your competition.Here's an exampl...(related: Search Engine Optimization)
Your Top Search Rankings At Yahoo & Msn Search Are Worthless
It seems I now do this rant every single year when traffic statistics are discussed in web industry news. Last week I posted to my Reality SEO blog tha...(related: Search Engine Optimization)
Duplicate Content Penalty - How To Lose Google Ranking Fast!
Duplicate content penalty. Ever heard of it? This penalty is applied by Google and possibly other search engines when content found on your website is largely the same as what is found elsewhere on your site or on other websites across the internet.Search engine spam has been common ever since search engines were first invented. Search engine spam describes the practice of making changes to your website that gets you listed high in search engines at the expense of readability by humans. Years ago, you could get ranked high on a search term simply by repeating it as many times as possible in a document. The primitive search engines of the past ranked the importance of a keyword simply by co...(related: Search Engine Optimization)
Link Building For Hilltop
Hilltop is one of the major concepts underpinning Google's search algorithm, yet its workings and implications are often misund...(related: Search Engine Optimization)
site-map - Copyright © 2006 Empower! Web Design | All Rights Reserved. | Search Engine Optimization