Network Performance Issues -- Friday, January 26, 2007, TT--59105

[Elwood J. Downing]

Merit Network staff regret to report some Members and Affiliate Members 
may have experienced intermittent service issues last Friday, January 
26, 2007 beginning around 11:30 a.m.

The issue ultimately came down to a card in an Ethernet switch in 
Chicago failing to forward received traffic on its gigabit ports.  The 
effect of this was difficult to diagnose, however, since in the interest 
of routing around single card failures, grouped services are mixed 
between port cards.   The net result was that some packets were being 
blackholed.

We observed significant packet loss which appeared to be attributable to 
one of three 1G circuits in Chicago.  We pulled that circuit, but the 
remaining congestion was unexpected and problematic.  Further 
sectionalizing identified one specific switch port.  We routed around 
it, and on Saturday reset the card.

Merit is in the process of installing a second switch site in Chicago as 
well as eliminating the bonding of 1Gs, moving to 10G cards where feasible.

On behalf of all Merit staff, we apologize for any inconvenience this 
service issue may have caused your organization. We at Merit 
continuously strive to provide the highest level of service to our 
Members and Affiliate Members and regret this service issue.

If you have any questions about this network event, please contact your 
support team at http://www.merit.edu/members/


Regards,
-- Elwood

---------------------------------------------------------------------------
Elwood J. Downing                      	e-mail: ejd@merit.edu
Merit Network                             Phone: (734) 936-2040
Member Services, Manager 			Fax: (734) 647-3185
    "The Buddhists say, when the student is
        ready, the teacher will appear."