Network telescopes collect and record unsolicited Internet-wide traffic destined to a routed but unused address space usually referred to as “Darknet” or “blackhole” address space [1]. Darknets can provide global perspective on Internet behavior and are one of the key data sources used by the networking and security communities to understand malware propagation [2–6], Distributed Denial of Service (DDoS) attacks [1, 7], network scanning [8, 9], routing misconfigurations [10], and Internet outages [11, 12]. Merit Network has been operating one of a small number of researcher-accessible network telescopes for more than 15 years, that has facilitated an array of empirical studies [2, 3, 9, 10, 13–21], to name a few.
Figure 1. Scanning and backscatter traffic captured in the Darknet.
Figure 1 illustrates the two basic types of Darknet traffic captured by a network telescope, namely 1) scanning and 2) backscatter. Note that Darknet traffic is unidirectional, i.e., the Darknet is completely passive and does not respond back. Next, we explain in some detail the origins and different types of scanning and backscatter.
Scanning arises as a result of multiple network activities. First, scanning includes reconnaissance activities from nefarious users or nation states that try to enumerate potential networking vulnerabilities of all Internet-accessible hosts. In their attempt to scan or “probe” the entire IPv4 address space they will eventually hit the dark IP space monitored by Merit’s Darknet team; as a result, the scanning packets sent by these nefarious users get recorded in PCAP format (i.e., a binary format for encoding information for network packets). The PCAP data thus provide valuable information such as the origin IP of the scanning activity, the port (i.e., application) that was being targeted, the exact time, duration and intensity of the scanning activities, etc.
A second type of scanning activity includes malware that try to propagate and “infect” more victims. Malware refers to malicious code running on compromised Internet hosts which are usually members of large botnets (i.e., large groups of compromised hosts running the same malicious code and controlled remotely by a “command-and-control center”); an example of such large botnet is the Mirai botnet [2], first appeared in late 2016 and still in operation. Similar to the Internet-wide reconnaissance activities discussed earlier, the malware also attempt to enumerate the whole IPv4 space to discover and infect new botnet victims. Inevitably, they would also fall into the Darknet “trap” and their network actions would also be recorded in PCAP format.
There are also benign types of scanning captured in network telescopes. These usually include “research scanning” activities such as the ones undertaken by Censys.io or Shodan.io . These organizations (and many others, including several universities) constantly scan the whole IPv4 space and interact with several applications/ports aiming to find misconfigurations and/or unsecure network hosts. This information is useful for understanding the Internet ecosystem and assessing the hygiene and security posture of various organizations.
Misconfigurations may also contribute to unsolicited Internet traffic appearing in a Darknet. This might be due to software bugs or inaccurate typing (i.e., “fat-fingered” typos) of network addresses. See the works of Wustrow et al. [21] and Benson et al. [15] for some illustrative examples.
Backscatter traffic captured in the Darknet represents a completely different network event. Backscatter traffic reveals victims of Distributed Denial of Service (DDoS) attacks and was first studied by Moore et al. [22]. Recently, Jonker et al. [7] examined backscatter traffic from a large network telescope and identified millions of DDoS victims. As Figure 1 depicts, backscatter emerges as a consequence of randomly-spoofed-based attempts to overwhelm a targeted host with heavy traffic in order to render them incapable of servicing their normal users. The attackers try to conceal their identity by selecting, in a random manner, a different IP from their true one. They then transmit large volumes of traffic to the victim IP using the falsified source IP. If the falsified IP happens to belong within the dark IP space monitored by the network telescope, the response from the victim to that (spoofed/falsified) IP would reach the Darknet and get recorded in PCAP format. Hence, the Darknet can capture exactly the time and identity of the attacked host.
Figure 2. The ORION Network Telescope. PCAP Darknet data are processed using a Go-based parser that extracts meaningful events (such as scanning and backscatter) and exports them, after being annotated with several auxiliary datasets, to Google’s BigQuery.
The architecture of Merit’s network telescope is depicted in Figure 2. The Merit Darknet currently consists of 1856 /24s subnets (i.e., around 500,000 dark IPs). The ORION (Observatory for cyber-Risk Insights and Outages of Networks) infrastructure— designed and engineered with support from the National Science Foundation [23]—receives and records Darknet packets in PCAP format, as explained earlier. Packets are written to disk on a continuous, non-stop, real-time basis and organized in hourly PCAP files.
The hourly PCAP files are then processed using a single-pass algorithm developed in Go by our team [24] to extract Darknet events, such as scanning and backscatter. The code adopts the methodology of Durumeric et al. [9] for defining events based on 1) the source IP of a packet appearing in the Darknet, 2) the port targeted and 3) fields from the transport protocol in the cases of TCP/UDP or fields from the ICMP header in case of ICMP. (All other protocols constitute a negligible fraction of Darknet traffic and for now are just grouped together.) An event essentially corresponds to a grouping of packets pertaining to the same activity (e.g., a scanning IP that targets a specific port and protocol) into a single record kept into our software’s memory (cache) until it “expires” and written on disk. We follow the approach of Moore et al. [1] to expire events that do not send any packets after 627 seconds; this interval corresponds to the “typical longest gap” (see [1], Section III.E, “Flow Timeout Problem”) expected to occur for an event lasting 2 days and having a target rate of 100 pps. Waiting for this specific timeout interval before we expire an event ensures that we will not split a sequence of packets belonging to the same event into two separate events or more.
All events are annotated with several summary statistics and other metadata, including the exact times the activity started and ended, the total bytes/packets transmitted into the Darknet, DNS information of the IP the event is associated with, geolocation information for the event’s IP, routing and ASN information, and others. These records are then uploaded to Google BigQuery (see Figure 2); for details regarding our BigQuery table schema please refer to our Github wiki [24]. Making the Darknet events available via BigQuery enables ease of data sharing, further analyses with standard SQL queries and integration with other data sources also in BigQuery such as data from Censys.io and others.
[1] D. Moore, C. Shannon, G. Voelker, and S. Savage. Network Telescopes: Technical Report. Technical report, Cooperative Association for Internet Data Analysis (CAIDA), Jul 2004.
[2] Manos Antonakakis, Tim April, Michael Bailey, Matt Bernhard, Elie Bursztein, Jaime Cochran, Zakir Durumeric, J. Alex Halderman, Luca Invernizzi, Michalis Kallitsis, Deepak Kumar, Chaz Lever, Zane Ma, Joshua Mason, Damian Menscher, Chad Seaman, Nick Sullivan, Kurt Thomas, and Yi Zhou. Understanding the mirai botnet. In 26th USENIX Security Symposium (USENIX Security 17), pages 1093–1110, Vancouver, BC, 2017. USENIX Association.
[3] Michael Bailey, Evan Cooke, Farnam Jahanian, Jose Nazario, and David Watson. The internet motion sensor: A distributed blackhole monitoring system. In Proceedings of Network and Distributed System Security Symposium (NDSS 05, pages 167–179, 2005.
[4] D. Inoue, M. Eto, K. Yoshioka, S. Baba, K. Suzuki, J. Nakazato, K. Ohtaka, and K. Nakao. nicter: An incident analysis system toward binding network monitoring with malware analysis. In 2008 WOMBAT Workshop on Information Security Threats Data Collection and Sharing, pages 58–66, April 2008.
[5] D. Moore, V. Paxson, S. Savage, C. Shannon, S. Staniford, and N. Weaver. Inside the slammer worm. IEEE Security Privacy, 1(4):33–39, July 2003.
[6] C. Shannon and D. Moore. The spread of the witty worm. IEEE Security Privacy, 2(4):46–50, July 2004.
[7] Mattijs Jonker, Alistair King, Johannes Krupp, Christian Rossow, Anna Sperotto, and Alberto Dainotti. Millions of targets under attack: A macroscopic characterization of the dos ecosystem. In Proceedings of the 2017 Internet Measurement Conference, IMC ’17, pages 100–113, New York, NY, USA, 2017. ACM.
[8] A. Dainotti, A. King, K. Claffy, F. Papale, and A. Pescap. Analysis of a /0; stealth scan from a botnet. IEEE/ACM Transactions on Networking, 23(2):341–354, April 2015.
[9] Zakir Durumeric, Michael Bailey, and J. Alex Halderman. An internet-wide view of internet-wide scanning. In Proceedings of the 23rd USENIX Conference on Security Symposium, SEC’14, pages 65–78, Berkeley, CA, USA, 2014. USENIX Association.
[10] Jakub Czyz, Kyle Lady, Sam G. Miller, Michael Bailey, Michael Kallitsis, and Manish Karir. Understanding ipv6 internet background radiation. In Proceedings of the 2013 Conference on Internet Measurement Conference, IMC ’13, pages 105–118, New York, NY, USA, 2013. ACM.
[11] Alberto Dainotti, Roman Amman, Emile Aben, and Kimberly C. Claffy. Extracting benefit from harm: Using malware pollution to analyze the impact of political and geophysical events on the internet. SIGCOMM Comput. Commun. Rev., 42(1):31–39, January 2012.
[12] Alberto Dainotti, Claudio Squarcella, Emile Aben, Kimberly C. Claffy, Marco Chiesa, Michele Russo, and Antonio Pescape. Analysis of country-wide internet outages caused by censorship. In Proceedings of the 2011 ACM SIGCOMM Conference on Internet Measurement Conference, IMC ’11, pages 1–18, New York, NY, USA, 2011. ACM.
[13] M. Bailey, E. Cooke, F. Jahanian, A. Myrick, and S. Sinha. Practical darknet measurement. In 2006 40th Annual Conference on Information Sciences and Systems, pages 1496–1501, March 2006.
[14] Michael Bailey, Evan Cooke, Farnam Jahanian, Niels Provos, Karl Rosaen, and David Watson. Data reduction for the scalable automated analysis of distributed darknet traffic. In Proceedings of the 5th ACM SIGCOMM Conference on Internet Measurement, IMC ’05, pages 21–21, Berkeley, CA, USA, 2005. USENIX Association.
[15] Karyn Benson, Alberto Dainotti, kc claffy, Alex C. Snoeren, and Michael Kallitsis. Leveraging internet background radiation for opportunistic network analysis. In Proceedings of the 2015 ACM Conference on Internet Measurement Conference, IMC ’15, pages 423–436, New York, NY, USA, 2015. ACM.
[16] Jakub Czyz, Michael Kallitsis, Manaf Gharaibeh, Christos Papadopoulos, Michael Bailey, and Manish Karir. Taming the 800 pound gorilla: The rise and decline of ntp ddos attacks. In Proceedings of the 2014 Conference on Internet Measurement Conference, IMC ’14, pages 435–448, New York, NY, USA, 2014. ACM.
[17] Alberto Dainotti, Karyn Benson, Alistair King, kc claffy, Michael Kallitsis, Eduard Glatz, and Xenofontas Dimitropoulos. Estimating internet address space usage through passive measure- ments. SIGCOMM Comput. Commun. Rev., 44(1):42–49, December 2013.
[18] Yang Liu, Armin Sarabi, Jing Zhang, Parinaz Naghizadeh, Manish Karir, Michael Bailey, and Mingyan Liu. Cloudy with a chance of breach: Forecasting cyber security incidents. In 24th USENIX Security Symposium (USENIX Security 15), pages 1009–1024, Washington, D.C., 2015. USENIX Association.
[19] A. Mirian, Z. Ma, D. Adrian, M. Tischer, T. Chuenchujit, T. Yardley, R. Berthier, J. Mason, Z. Durumeric, J. A. Halderman, and M. Bailey. An internet-wide view of ics devices. In 2016 14th Annual Conference on Privacy, Security and Trust (PST), pages 96–103, Dec 2016.
[20] Matthew Sargent, Jakub Czyz, Mark Allman, and Michael Bailey. On the power and limitations of detecting network filtering via passive observation. In Passive and Active Measurement, volume 8995 of Lecture Notes in Computer Science, pages 165–178. Springer International Publishing, 2015.
[21] Eric Wustrow, Manish Karir, Michael Bailey, Farnam Jahanian, and Geoff Huston. Internet background radiation revisited. In Proceedings of the 10th ACM SIGCOMM Conference on Internet Measurement, IMC ’10, pages 62–74, New York, NY, USA, 2010. ACM.
[22] David Moore, Geoffrey M. Voelker and Stefan Savage, Inferring Internet Denial-of-Service Activity, USENIX Security, 2001.
[23] Michalis Kallitsis, Zakir Durumeric, Stilian Stoev, CNS-1823192, CRI: II-New: ORION: Observatory for Cyber-Risk Insights and Outages of Networks.
[24] Conrad Edwards, Mark Weiman, Zakir Durumeric, Michalis Kallitsis, “Go-based Darknet parser to extract Darknet events”, https://github.com/Merit-Research/darknet-events
Project Support:
- NSF CRI: II-New: ORION: Observatory for Cyber-Risk Insights and Outages of Networks. Click here.
- CCRI: NEW: CLASSNET: Community Labeling and Sharing of Security and Networking Test datasets. Click here.
- CLASSNET project portal: Click here.
Software and Documentation:
See our code and wiki for more information on the data generation, the schema employed and examples of BigQuery-driven data analysis.
Project collaborators:
University of Michigan; Stanford University; Penn State University, University of Southern California; University of California, San Diego / CAIDA; Georgia Institute of Technology.