North American Network Operators Group|
Date Prev | Date Next |
Date Index |
Thread Index |
Author Index |
Potentially on-Topic: is MSNBot for real?
- From: Mark Foster
- Date: Fri Sep 22 06:42:00 2006
On a website I host with nearly 9000 unique visits month-to-date (thats
visits, not hits) a full 20% of the recorded 'hits' (Hitcount is ~40,000)
are being generated by 'msnbot'. We see this as a large amount of http
traffic from IP addresses owned by Microsoft.
I've actually seen this across a number of websites (including my own) but
the guest on my server has raised the issue of loading being completely
misproportionate to the perceived value of the visit - and asked about
potentially blocking them off entirely.
Is this unusual, or what? Are search engines supposed to be amongst the
biggest user agents recorded on a typical website? How much trolling and
indexing is considered 'too much' ?
At what point to the search engines themselves become a menace - the load
the cause outweighs the value of said load? (I'd like my cpu cycles to be
for real people, please...)
Off-list thoughts on this welcome if the operational relevance of this
issue is questioned...