Merit Network
Can't find what you're looking for? Search the Mail Archives.
  About Merit   Services   Network   Resources & Support   Network Research   News   Events   Home

Discussion Communities: Merit Network Email List Archives

North American Network Operators Group

Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical

Potentially on-Topic: is MSNBot for real?

  • From: Mark Foster
  • Date: Fri Sep 22 06:42:00 2006


On a website I host with nearly 9000 unique visits month-to-date (thats visits, not hits) a full 20% of the recorded 'hits' (Hitcount is ~40,000) are being generated by 'msnbot'. We see this as a large amount of http traffic from IP addresses owned by Microsoft.

I've actually seen this across a number of websites (including my own) but the guest on my server has raised the issue of loading being completely misproportionate to the perceived value of the visit - and asked about potentially blocking them off entirely.

Is this unusual, or what? Are search engines supposed to be amongst the biggest user agents recorded on a typical website? How much trolling and indexing is considered 'too much' ?

At what point to the search engines themselves become a menace - the load the cause outweighs the value of said load? (I'd like my cpu cycles to be for real people, please...)

Off-list thoughts on this welcome if the operational relevance of this issue is questioned...

Cheers
Mark.




Discussion Communities


About Merit | Services | Network | Resources & Support | Network Research
News | Events | Contact | Site Map | Merit Network Home


Merit Network, Inc.