Since 1987, Merit has been providing superior front-line support for its Members, first with the network operations center (NOC) and currently with its Merit Support Center. The speed and technology of networking has changed significantly during the last 25 years, moving from T-1 and T-3 connections to high-speed gigabit fiber connections. The tools, support and services offered by Merit’s support personnel have changed as well, but the goal of providing the best support and service has remained a constant.
Merit’s first network operations center (NOC) was based in a small room in the University of Michigan’s Computer Center Building. In 1987, it was staffed by three people and operated during regular business hours. The NOC monitored the PCP (Primary Communications Processor) and SCP (Secondary Communication Processor) machines on Merit’s network and the local network at the University of Michigan. Staff used a paper note card filing system to keep track of changes to the processing machines.
After Merit Network was awarded the NSFNET contract, a larger NOC was built in U-M Computer Center Building. The physical layout of the NOC facility was modeled after the operations centers that phone companies used, except that the new NOC would monitor an Internet Protocol (IP) network. The staff grew to 10 operations personnel and two developers. The NOC facility became functional when the NSFNET began operations in July 1988.
To monitor the NSFNET, Bill Norton, a NOC developer at the time, created a monitoring tool called Internet Rover, which was intended to be an interim step to monitor the NSFNET backbone while IBM and Merit staff worked to get a large mainframe network management system up and running. Internet Rover worked so well that it was adopted as the primary monitoring tool, even after the large network management system was completed.
“As typically happens, the interim solution was stepwise refined over years and ultimately became the primary solution,” said Bill Norton, who was a co-founder of Equinix in 1998 and is now executive director at DrPeering International. “I made it available to the public as freeware and others picked it up and used it to monitor the CIA Network, the Italian national network, a wide set of universities, etc. I guess free is a good price, and all the step wise refinement done seemed to create a tool custom suited to the Internet Network Operations Center.”
“One key benefit of the tool was that you could have an almost unlimited number of display screens showing the alerts, but only one collector of network information. That way the NOC, the engineering team, help desk, etc. could all see the alerts and their status,” Norton continued. “The NOC knew the state of the network within a few minutes even when things were really broken.”
The developers also created several other helpful tools:
- MeritRover – monitored the status of the old PDP-11 nodes across Michigan.
- Pingky – a parallel pinger that determined network reachability and created a file when the reachability test failed and removed the file when it succeeded.
- IntetRover – a program that tested various services like mail, DNS, etc. and also checked for the existence of a file created by pingky. This program created alerts and removed them.
- Display – was the alert display screen that the NOC operators used that allowed people to update alerts with status messages
The network operations center was based in the U-M Computer Center Building for 21 years, and in March 2008, the NOC moved into its current location in Merit’s offices in Ann Arbor. The NOC continued to provide monitoring and support for Merit’s network, 24 hours a day, seven days a week.
In 2009, Merit added the Services Desk to provide dedicated support for MeritMail. Services Desk staff offered technical support to Merit Members during regular business hours.
As Merit’s services portfolio and the number of Merit Members outside of Michigan grew, it became apparent that there was a need for 24-hour support and greater resources dedicated to Merit’s services. Since the NOC had been providing 24-hour support and monitoring for Merit’s network for many years, the decision was made in 2010 to merge the NOC, the Services Desk, and Member Relations support staff to create the Merit Support Center (MSC).
“The Merit Support Center is here to serve Merit Members and the Merit community, to provide the best support and service possible,” said Riva Milliken, director of the Merit Support Center.
The MSC now monitors and supports Merit’s statewide network and a wide portfolio of services, including Merit Cloud Media, Merit Cloud Storage, MeritMail, Merit VirtualDataCenters, MeritVoice, and many others. In addition, the Merit Support Center assists with new service inquiries and provides E-rate support. The support center is open around-the-clock, all year long.
The Merit Support Center facility has been designed to promote collaboration and to provide easy access to relevant network monitoring and service support information. MSC staff is stationed at several interconnected workstations, enabling them to consult with each other when needed. A large screen on a nearby wall provides continual updates and alerts to staff.
“By combining the talents of the Merit Services Desk, Network Operations Center, and Member Support coordinators, this MSC has realized increased efficiencies and effectiveness, enabling Merit to provide an extraordinary level of support and service to our membership. By supporting the entire Member experience from pre- to post-sale, the Merit Support Center is able to provide our membership with a comprehensive high value-add information resource as well as a key problem-solving engine,” stated Milliken.
“With the MSC providing presale through post-sale support, it’s given us a holistic view of the service. We have found some efficiencies. Everything is in-house, within the MSC, so we’ve gotten faster at things.”
Similar to past network operations center capabilities, the Merit Support Center uses specialized resources for monitoring, but with the addition of a custom email notification tool and processes for determining service issue priorities. Handprints, a tool was created in-house, debuted in September 2010 and provides alerts and ticketing, using data from Merit’s enterprise monitoring system.
“System alerts are sent to a screen monitored by MSC staff,” said Shane Burgess, a developer at Merit Network. “The MSC reviews the alerts, opens a ticket based on the information provided and then notifies the effected users.”
The system uses seven levels of service issue priorities, from class 1 (catastrophic) to class 7 (service request-standard), to triage maintenance, upgrade, and service requests. The MSC assigns a priority to an issue, collects the needed information, and then sends an email notification to each effected Member. The MSC resolves the issue, engaging second level support as needed. Members can access basic information about their service tickets information through the Merit Member Portal.
“The service levels have given us a common understanding, a shared language with others within Merit,” Milliken said. “Regardless of the service or issue, all of the team members within Merit have the same understanding when an issue is assigned a service level priority.”
“Handprints streamlines the service ticket information. The Merit Support Center can use it to generate reports based on effected Members, nodes, problem types, and how issues were resolved,” Burgess said.
“The MSC can follow our monitoring system and Handprints tickets on one screen. Within minutes, we can notify the necessary staff about service issues,” Milliken said.
Merit’s support, processes, and monitoring systems have changed over the years, but the commitment to providing Members with the best support possible has been a constant. Future goals for the Merit Support Center include providing live assistance and greater functionality through the Merit Member Portal, including the ability for Members to open and update service issue tickets.