Merit Network Downtime, Outages and other Events
Date Prev | Date Next |
Date Index |
Thread Index |
Author Index |
Historical
**UPDATE** TT60009: Merit Network Problems 08/20/07 18:45 EDT
- From: Network Operator
- Date: Mon Aug 20 19:01:37 2007
-----------------------------------------------------------------------
SUBJECT: Merit Network hardware failure.
SCOPE: Merit Regional Network.
STATUS: Internet traffic via Merit Network has been stable since
Saturday but we are keeping our ticket open while we
continue to work with our equipment vendors on a long
term solution.
CAUSE: After careful investigation over the the weekend Merit
believes the actual cause was an unstable 10G link
between our Chicago and Grand Rapids core switches. This
instability was causing this link to bounce erratically,
at some times as often as once a second, at other times
there was an hour or more between bounces. Each time
this link bounced the protected VLANs on the ring were
wrapping and unwrapping (i.e. changing directions ). Each
time the VLANS changed directions, notification messages
were sent between the switches. The vendor indicated that
a possible memory leak in the switch software in
combination with this large flow of notification messages
led to a depletion of available memory in the switches,
causing packet-forwarding issues. To stabilize the
network Merit has disabled the link between Grand Rapids
and Chicago while we repair the circuit.
START TIME: Thursday, 15 August 2007, 22:32 EDT
-----------------------------------------------------------------------
EAD
---------------------------------------------------------------------
UMnet Network Operation Center Merit
---------------------------------------------------------------------
University of Michigan Merit Network, Inc.
------------------------- --------------------------------
trouble@umich.edu trouble@merit.edu
www.itd.umich.edu/outages www.merit.edu/mail.archives
---------------------------------------------------------------------
|
