TT60009 -- Merit Network Backbone Service -- Update

[Elwood J. Downing]

We are pleased to provide an update to the recent backbone service outage. 
As reported last night, 9:29 PM, EDT, Thursday, August 16, 2007, Merit's 
backbone began to stabilize around 8:20 PM, EDT on Thursday and has 
continued without any additional network alerts or major problems reported 
to us from our Members.

We are continuing to work with our backbone equipment vendors to determine 
what caused the problem and how to prevent it from happening in the future. 
We believe that we will have enough information to share with you the "How, 
What, and Why's" of this network problem.

Please plan to join us via web and audio conference on Tuesday, August 21, 
2007 at 11 AM to Noon for a Merit Community Update where we will have more 
information about the problem and provide you with the ability to call in 
and ask questions. Merit engineers and management staff will be present 
during this broadcast to answer your questions and provide additional 
information.

The main cause of the problem seemed to be with the management interface 
cards on the Extreme Aspen 10G switches, which were crashing. We would 
reset the card and the system would work for a while then the card would 
crash again. This also became a problem with these cards that had 1G LAN 
ports on them servicing our members and connecting to routers. Some cards 
did not come back after resetting which required a hard reset (power 
cycle). Since we and Extreme Engineers did not know the nature of the 
problem, we were also investigating a possible Denial of Service attack on 
our network.

What we found is that our 10G ring was changing up/down states rapidly 
which made us think this was causing the hardware to stop working because 
they were running out of resources.  We then determined that the root of 
the problem was here in Ann Arbor, where a Cisco switch was connected to 
our 10G  core Extreme Networks switch. We disabled the port and the network 
began to stabilize. We have informed Extreme Networks and they are working 
to provide feedback on the problem and resolution. It took additional time 
for all the routes to propagate since many of our networks were route 
dampened because of the on-going instability of the network.

If you are continuing to experience any network performance problems please
contact Merit's Network Operations Center (NOC) immediately. We have 
engineering, NOC, and support staff available to work with you on resolving 
any issues you are experiencing.

We sincerely apologize for the inconvenience this outage has caused your 
organization. We continuously strive to provide the highest level of 
service to our Membership and regret this service issue.

If you have any questions about this network event, please contact your
support team at http://www.merit.edu/members/ for additional information
and assistance.

Sincere Regards,
--Elwood

---------------------------------------------------------------------------
Elwood J. Downing                      e-mail: ejd@merit.edu
Merit Network                          Phone: (734) 936-2040
Director of Member Services            Fax: (734) 647-3185

    Merit Network -- Connecting Organizations, Building Community