Re: Network Outage Webcast is now available -- And Additionalquestions addressed

[Elwood J. Downing]

Hi Folks:


Listed below are the additional questions that were asked during the 
webcast that we did not get to address on Tuesday. We are also working on 
providing the questions and reponses from the archived webcast link page as 
well. However, we wanted to get them out to you as soon as possible.

The archived webcast is available at: http://www.merit.edu/ -- in the lower 
right corner of the screen under "News:"; section click on the "Network 
Outage Update" link. You can also go directly to the Network Outage Update 
informational and webcast link page at: 
<http://www.merit.edu/news/newsarchive/article.php?article=20070822_network>

Regards,
--Elwood

---------------------------------------------------------------------------
Elwood J. Downing                      e-mail: ejd@merit.edu
Merit Network                          Phone: (734) 936-2040
Director of Member Services            Fax: (734) 647-3185

    Merit Network -- Connecting Organizations, Building Community


---start of questions and Merit's response ----
1. Name: Erich Flynn
Organization: GRCC

Question: Are you considering for your backup non-Extreme switches or other 
routers?


Merit's Response:

Yes.  Merit plans on installing a 10G wavelength from Ann Arbor to Chicago 
on a different Optical Transport system, as redundancy for the Extreme 
Aspen switches. This wave would connect directly between our Cisco CRS 
routers located in Ann Arbor and Chicago.   This is at no additional cost 
to Merit.  This wave, in combination with our Mid-Michigan fiber, adds 
considerable redundancy, however we are in the process of assessing what 
links are still vulnerable.


2. Name: Tom Davis
Organization: MSU

Question: When is the next fiber maintenance scheduled?

Merit's Response:

There was scheduled work in Chicago that has been canceled.  This afternoon 
Tuesday August 21, Level3 notified us that they planned on doing more 
maintenance tonight in Southeast Michigan.  We requested that the 
maintenance be cancelled, and Level3 could not do so, but said it was on 
fibers other than ours.  We will be closely monitoring this activity any 
future fiber maintenance announcements and correlating any subsequent 
observance of network change.

3. Name: Greg Vedders
Organization: Aquinas College

Question: Can you clarify what we should be experiencing  in Grand Rapids, 
we can currently get to other Merit sites including those on the East side 
of the state, but cannot get anywhere else on the Internet.  Whenever there 
is a network problem, it seems we never route through Detroit...what good 
are the redundant links if we can never use them?

Merit's Response:

Because of the route flapping of Merit's announcements, our BGP peers 
effectively stopped listening to them in both Chicago and Detroit.


4. Name: Bob Eluskie
Organization: Grand Rapids Community College

Question: We cannot get to the Merit network from the Sprint Wireless 
Broadband network.

Merit's Response:

Because of the route flapping of Merit's announcements, our BGP peers 
effectively stopped listening to them in both Chicago and Detroit.  Thus 
Sprint could not route to Merit.


5. Name: Jim Cotton
Organization: Western Michigan University

Question: This is not the first issue with the Extreme switches.  There 
have also been interfaces that dropped out of channels and other resource 
issues over the past six months.  Is the failure rate of the Extreme 
switches high enough to consider using a different vendor?


Merit's Response:

While we have had issues with Extreme switches in the past, these issues 
have for the most part been resolved with software upgrades.  The switches 
in this topology have functioned with a high degree of integrity for nearly 
two years.   The switches' failure in this instance was a multi-faceted 
problem related to an amplifier having shut itself down, as well as the 
normal functioning of EAPS (Ethernet Automatic Protected Switching). We are 
continuing to investigate the EAPS thresholds that perhaps need to be tuned 
as well as improved alert notifications and other aspects.


6. Name: Mark Driscoll
Organization: University of Michigan

Question: It appears that the resolution to these outages is to shut down 
affected amplifiers during a fiber maintenance.  Does Merit have the 
ability currently to shut these amplifiers down remotely, and the ability 
to turn them back up?

Merit's Response:

We would not necessarily want to shut the amplifier down since that was a 
cause of the problem in this instance.  We believe the better tactic would 
be to shut down the 10G circuit on a link where we knew maintenance was 
going to occur, and allow protected switching to preserve service 
integrity.  We did not have this conservative response to scheduled 
maintenance in our standard practices previously, but do so now.


7. Name: Peter Murphy
Organization: Wayne State University

Question: Please comment on how often we've had fiber maint that did NOT 
cause such problems.

Merit's Response:

There have been dozens of instances of maintenance on our 10G fiber 
network, including on this specific link,  that have had no effect on 
service performance.  We are investigating why this maintenance event had 
such an impact.



8. Name: Dale Karbginsky
Organization: Kettering University

Question: When do you expect things to return to normal. ETR so I can pass 
it on to our campus.

Merit's Response:


As of 12:15 pm 8/21/07 the network returned to normal status.

9. Name: William Wachob
Organization: Lawrence Technical University


Question: Do you think it is appropriate for Merit to stand-down on these 
maintenance activities by Level-3? Have the maintenance procedures been 
modified to include handling of the network amplifiers (shutting them down, 
restarting them and testing their performance)?

Is it possible to get an SMS message when the network is experiencing 
"significant" problems?

Merit's Response:

While most of this was addressed in previous questions, we confirm that we 
are taking a more proactive approach in dealing with fiber vendor 
maintenance procedures.

--- end of questions and response ----