Merit Network
Can't find what you're looking for? Search the Mail Archives.
  About Merit   Services   Network   Resources & Support   Network Research   News   Events   Home

Discussion Communities: Merit Network Email List Archives

North American Network Operators Group

Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical

Re: Crawler Ettiquette

  • From: Bob K
  • Date: Wed Jan 23 15:32:23 2002

On Wed, Jan 23, 2002 at 02:35:17PM -0500, Deepak Jain wrote:
[snip]
> This information will be made available to research institutions and
> other concerns.
[snip]
> 	c) Allow ISP's caches to sync with it.
[snip]
> ISPs who cache would have an advantage if they used the cache developed by
> this project to load their tables, but I do not know if there is an
> internet-wide WCCP or equivalent out there or if the improvement is worth
> the management overhead.
[snip]

Assuming that the info will be made available in html format, the only
thing you really need to do to achieve c) is to choose an appropriate
value for the http-equiv="Expires" meta-tag when serving the info, and
have a cron job at each ISP make a request for the info at some
arbitrary time.  This last step really isn't that useful unless there
are points of congestion, or times when the servers are bogged down.

The caches have to respect the Expires tag, though, and a broken clock
can cause all sorts of fun on that end...

-- 
Bob <melange@yip.org> | Please don't feed the sock puppet.




Discussion Communities


About Merit | Services | Network | Resources & Support | Network Research
News | Events | Contact | Site Map | Merit Network Home


Merit Network, Inc.