Merit Network
Can't find what you're looking for? Search the Mail Archives.
  About Merit   Services   Network   Resources & Support   Network Research   News   Events   Home

Discussion Communities: Merit Network Email List Archives

North American Network Operators Group

Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical

Re: Crawler Ettiquette

  • From: Bob K
  • Date: Wed Jan 23 15:40:31 2002

<embarassed look>

Er, I just re-read that, and now understand what you meant by c).
Sorry for the wasted bandwidth.

On Wed, Jan 23, 2002 at 03:29:38PM -0500, Bob K wrote:
> 
> On Wed, Jan 23, 2002 at 02:35:17PM -0500, Deepak Jain wrote:
> [snip]
> > This information will be made available to research institutions and
> > other concerns.
> [snip]
> > 	c) Allow ISP's caches to sync with it.
> [snip]
> > ISPs who cache would have an advantage if they used the cache developed by
> > this project to load their tables, but I do not know if there is an
> > internet-wide WCCP or equivalent out there or if the improvement is worth
> > the management overhead.
> [snip]
> 
> Assuming that the info will be made available in html format, the only
> thing you really need to do to achieve c) is to choose an appropriate
> value for the http-equiv="Expires" meta-tag when serving the info, and
> have a cron job at each ISP make a request for the info at some
> arbitrary time.  This last step really isn't that useful unless there
> are points of congestion, or times when the servers are bogged down.
> 
> The caches have to respect the Expires tag, though, and a broken clock
> can cause all sorts of fun on that end...
> 
> -- 
> Bob <melange@yip.org> | Please don't feed the sock puppet.

-- 
Bob <melange@yip.org> | Please don't feed the sock puppet.




Discussion Communities


About Merit | Services | Network | Resources & Support | Network Research
News | Events | Contact | Site Map | Merit Network Home


Merit Network, Inc.