Merit Network
Can't find what you're looking for? Search the Mail Archives.
  About Merit   Services   Network   Resources & Support   Network Research   News   Events   Home

Discussion Communities: Merit Network Email List Archives

North American Network Operators Group

Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical


  • From: Frank Scalzo
  • Date: Wed Aug 28 23:25:22 2002

Whoops! 2 hours to find routers w/o an IGP tsk tsk.

Dear AT&T IP Services Customer,

 Please be advised of the following:

  Date:                          8/28/02
 Customer Care TT#:      1070909
 Start of Impairment:    14:56 ET
 End of Impairment:      16:52 ET

 On behalf of AT&T, we would like to extend our apologies for any inconveniences to your business caused by the
 service impairment and delay in restoring your service.

 AT&T IP Services customers with traffic routed through our Chicago backbone routers may have been affected by
 this impairment.

 Our Network Engineers found that OSPF* network statements were missing from two backbone routers, causing the
 routing issues that customers were experiencing.  AT&T Network Engineers manually reloaded the OSPF network
 statements and then proceeded to reboot the routers in order for the changes to take effect. 

 The network statements were mistakenly deleted during a routine configuration update to our backbone routers. 
 Management has been made aware of these findings and appropriate actions will be taken to ensure this situation
 does not re-occur.

 Prior reported information was leading the AT&T Network Engineers to believe that there were issues with the route
 reflectors being out of synch.  However, this was only symptomatic of the problem stated above.  In other words, due
 to the missing network statements, the route reflectors were not advertising certain networks; therefore traffic was
 not properly routed.

 It is AT&T’s goal to provide the highest level of service to our customers. 
 We appreciate your business and look forward to continuing our relationship in the future.

 *Routing protocol (Open Shortest Path First)

 Depending on the particular IP access services to which you are subscribed you may also find additional information
 as it becomes available at:



 If you require further information, please feel free to contact AT&T at:

 AT&T MIS: 1-888-613-6330
 AT&T VPNS: 1-888-613-6501

 Thank you for using AT&T.


 The AT&T Customer Care Team

-----Original Message-----
From:	Matt Levine []
Sent:	Wed 8/28/2002 6:21 PM
To:	Mike Tancsa
Cc:	Wes Bachman;
Subject:	Re: AT&T NYC

On Wednesday, August 28, 2002, at 04:17 PM, Mike Tancsa wrote:

> is not currently reachable, but AS15290's 
> router server is for those who want a view on things...

Interestingly enough, ATT is announcing to BBN (and nobody 
else, including AS7018 internal)..

> and
> which come in handy :-)
>         ---Mike
> At 04:11 PM 28/08/2002 -0400, Mike Tancsa wrote:
>> I am seeing this as well. One of my upstreams (AT&T Canada- 15290) 
>> has connections with AT&T US (7018) in Chicago and Vancouver.  
>> Chicago seems to have disappeared for me and all traffic bound via 
>> that path is going via Vancouver now.
>>         ---Mike
>> At 02:52 PM 28/08/2002 -0500, Wes Bachman wrote:
>>> Bryan,
>>> There is a known AT&T outage in Chicago currently.  Could this be
>>> effecting you in some way?
>>> -Wes
>>> On Wed, 2002-08-28 at 14:44, Bryan Heitman wrote:
>>> >
>>> > Anyone seeing any problems with ATT in new york?
>>> >
>>> >
>>> > Best regards,
>>> >
>>> >
>>> > Bryan Heitman
>>> > Interland, Inc.
>>> --
>>> Wes Bachman
>>> System & Network Administration, Software Development
>>> Leepfrog Technologies, Inc.
Matt Levine
ICQ  : 17080004
AIM  : exile
GPG  :
"The Trouble with doing anything right the first time is that nobody
appreciates how difficult it was."  -BIX

Discussion Communities

About Merit | Services | Network | Resources & Support | Network Research
News | Events | Contact | Site Map | Merit Network Home

Merit Network, Inc.