Legitimate-Bots:
The Legitimate-bot analysis page shows list of bots that have crawled through your site. Legitimate-bots do not have any malicious intent and some legitimate-bots tend to bring traffic to your site.
Legitimate-bots are classified into 5 categories to let you take necessary action against them.
- Monitoring Bots - Bots which are used to monitor the system health of the customer's websites. (e.g. Pingdom)
- Back link checker bots - These categories of bots check the back links of the URLs. (e.g. UASlinkChecker)
- Social Network Bots - Bots which are run by social network sites. (e.g twitterbot)
- Partner bots - Partner bots which are useful to the website (e.g. Paypal IPN). Ad networks and SEO bots are also included.
- Aggregator bots - Bots which collate information from other websites (e.g. WikioFeedBot) . Market intelligence purpose crawlers are also included.
If you are concerned about the server load because of the crawling of these legitimate-bots, you can block the categories which are not useful to your site. To take action, open “Bot Response List” page on the portal and select the required action against the legitimate-bot categories.
You can get insights on Legitimate bots by selecting Legitimate-Bot analysis page. List of all Family names of bots hitting this site are shown. Corresponding category and number of hits from that family are shown.
Clicking on “View IP Details” shows all the IP addresses under that family name which are hitting the site and their hits number. For example, vebot IP details are shown as below. You can close the details by clicking on X at the top right corner.
Search Engine Bots:
The search engine bot analysis page shows the list of good crawlers such as Google, Bing, Yahoo etc., that have crawled through your site. These bots bring traffic to your site.
