North American Network Operators Group|
Date Prev | Date Next |
Date Index |
Thread Index |
Author Index |
2006.06.07 NANOG-NOTES Lightning talk notes
- From: Matthew Petach
- Date: Fri Jun 09 18:50:56 2006
- Domainkey-signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:sender:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition:x-google-sender-auth; b=oTQWq8lHGY14GZIZtrNL6n6LcAqHMMDXdtPFq/Y3iJxcYeOjw15rSXMMXAdjeH4a2btvMC2+1VGow6j5AEOOEUuUM6Xl7Jy4cTELOWNykd8AMUDtuB67R+kLBBLh1b5goaP281ZKxcNgVGnJL6+QVMnLODb3Nn8WT8pybvosYKc=
(I think these were the toughest to take notes on, since they went
by so fast; took the most cleaning up afterwards. But they were also
the best talks of the 3 days. I wish we could have flipped, and taken
more time on Tuesday for them so we really could have dug in and
asked the questions we were itching to ask. ^_^; --Matt)
2006.06.07 Lightning talks
Marty Hannigan, Renesys:
[slides are at:
Critical infrastructure, root server location
Where to stick your servers. :)
he took some public info out there on root-servers.org
talked to some people, extrapolated from larger set of
3 corp a, c, j
2 edu b and d
1 mil g
2 research e/h
3 nonprofit f, i, l
autonomica is responsible for l, but hosts "some"
instances on a CDN; CDN is a US formed entity
1 non profit k
1 nonprofit m
92% of system operated in US, 8% non-us;
5% margin of error +-.
US entity type
us corp 39%
us mil 23%
us edu 15%
us nonprofit 15%
in 54 countries
all methods of governance
79% are democratic governments
21% in other forms of government
global diversification for security and performance
instances spread across continents
weaknesses become strength, since they are diverse;
no one weakness knocks out all servers.
little less open to insider malfeasance
east EU 3%
getting reasonable coverage in the world
situating a root server
who you know
ICANN, operator, IX, and RIR relationships
how you spin it
performance and security
betterment of user experience
no different from anyone else
miscreants masking other activities
Not sure what motivations to attack root servers;
can't extort money from nonprofits
let's attack a root server
location; eu hosting facility
multi-post cabinet config with cabling and power
unlocked cabinet, single factor facility entry
open cabinet door
access to power
advertise a route
return bad answers
random host queries
root system is less likely to be subject to insider
attack or weakness
but can be attacked by layer 3
there is likely good resarch data coming across those
trend towards a collapsed root system, where root and
TLD share same hardware or networks should be more
slides will be up soon, talk to him in the hallway
NEXT, Anton Kapela
[slides are at:
I'm pinging 10: high rate active probes
we're pinging stuff really quickly
adjusted host kern.hz to 1000 select() gets pretty
accurate +-1ms emmission accuracy
stuff is responding
Interesting 0.001% of data relates to end-to-end queuing
what has been sampled?
some cisco 7513s
IOS 12.3 mainline
various end-to-end paths on u-wisc network
raw data isn't terrible interesting.
in adaptive link layer protocols, see rate shifting
manifested in RTT
wireless, HPNA/HCNA, powerline ethernet
10,30,60,90 second peaks
fourier transforms, wavelet transforms, frequency domain
1000 seconds at 10ms intervals
break into composite, aggregate graph at top,
0-50hz span on x axis, y axis is contribution
summary of entire graph.
bottom right graph is rough 200 samples of a
range from 0-5hz, 100pps, deduce delay at half
that sampling rate.
delay is not a simple boring thing; has
scheduler delays, path dynamics not visible
before to see queue depths.
shark fins showed up; congestion events do
occur, are quite measurable.
when links are hot, queues are obvious, esp. on
highly multiplexed links.
bottom left, cubic resonance, several tens of
thousands of multiplexed flows hitting odd
pinging windows machine, composite spectral
fingerprint; 10,20,25,30 spikes
Linux fewer spikes
freebsd low and flat
IOS is 10, 20, 30 and grass of 1hz spacing
win32 delay spectrum also has 1hz fuzz below
Sampled RTT and performed signal analysis of it;
is network time continuous? is round trip time
discreet or continous?
no changes in revealed as you go down lower
is delay a "signal' anyway
what's with the 0 hz DC component in the FT output?
could this be used for fingerprinting?
yes, could be like next nmap.
packet-level fingerprinting is trivial to fake; but
IP stack scheduler behaviour doesn't change so
Affect on traffic from the TPB bust
with Kurtis Lindqvist
[slides are at:
p2p protocol for filesharing.
text string, upload to tracker, get IPs of other clients
that have done the same thing, clients connect to each
other, develop a swarm.
clients communicate even when tracker vanishes.
just can't get new clients joining
run by a handful of individuals aged 22 to 28
used ~100mb at peak
peaked at 2M concurrent users
stats code in tracker indicated that total p2p
traffic was close to 100gig/sec
thus far, largest bittorrent site/tracker in world
photo slide showing the physical gear
10 high-end small servers in half a rack in stockholm,
web frontends, db servers, trackers
on the stats
not an exact science
at least a german ISP had an outage at the same time
bust was around 12.00 CET may 31st (euro time)
data collected from Euro-IX members
some saw no difference.
Netnod aggregated, biggest drop, about 10+Gb drop
in Netnod stockholm *very* visible.
stats server was slashdotted, lost an hour of stats.
LINX London, saw about 5Gb drop out of 80Gb
AMSIX dropped about 5Gb out of 160Gb
DECIX frankfurt, germany, drop before noon,
FCIX, helsinki, Finland
drop fairly visibile
NIX, in norway, drop also visible.
doesn't show private exhanges/private peerings
Brussels (BNIX) also saw drop.
netflow export from big US ISP,
large chunk of bittorrent traffic packets faded off.
Thepiratesbay.org was back online 72 hours later in
and traffic started coming back
June 6th is a holiday, watch the stats this coming
Police took ALL hosted equipment at the same site
by the same hosting company (small one, only a few
racks), caused quite a few community web sites to
go down plus commercial customers
Has spawned a lot of discussion in Sweden regarding
all issues involved. Front page material every
day, even video surveillance of the raid from
surveillance cameras has been posted on youtube.com
Accusations of police/politicians being influenced
by White House and MPAA and others
Q: Bill Norton: what about other tracker sites, why
didn't traffic just shift to them?
A: some did, but torrent files have the tracker hard
coded in them, so they can't just flip over to other
tracker sites on their own.
Q: Roland Dobbins, back up in several countries now
including Russia, is traffic back?
A: Keep watching the graphs.
And if you want to see the bust, search for
"pirate bay" and "police", there's one link on youtube.
[slides are at:
Passive Metro WDM
how it works
single mode fiber: mutiple wavelengths
also called "colours" or "lambdas"
pluggable optics as enabler
low cost for passive optical equipment, particularly
Dark fiber IRUs are very cheap.
how does it work?
O band Original 1260 - 1360
E band Extended 1360 - 1460
S band Short 1460 - 1530
C band Conventional 1530 - 1565
L band Long 1565 - 1625
active WDM cisco 15xxx, cienna, movaz, others
passive WDM using optical filters
self-assembled patch panels
complete systems (CUBO)
pictures of components
wideband multiplexing (1350/1550)
2GE fdx per pair, 1 GE fdx per strand
single strand networking the receiver is *always*
low cost for transcievers (LX/ZX, <$500)
10GE possible (ER/LR)
beyond this scope
everyone knows how to do it, it just costs more.
wavelength, wide channels, 8 channels
1270-1470 low range
cost is cheap ~$1000 per strand per end for
$300-$1000 per GBIC depending on quality
(CUBO, Taiwanese hw manufacturers)
no Xenpaks, GBICs only
20nm channel spacing
low availability on 'low range' GBICs/SFPs
each channel is narrow
0.8nm == 24 dense channel per single coarse channel
160 channels easily
research at 12.5Ghz
Xenpaks available $9k+
few GBICs at $1500+
build/add as you grow by mixing and matching
available in various ranges (center wavelength,
Going from GWDM to GWDM/CWDM to GWDM/CWDM/DWDM
Testing and management
optical power meter
communication is key
OOB access: HOOTS, cell phone
you need to talk site-to-site to coordinate
make sure cell phones don't depend on fiber
optical power monitoring/APD receivers in GBICs
(show interface blah trans)
few complete commercial systems available
systems require clue and duct tape to put together
need to tune with attenuators if signal is too
strong, attenuators differ with wavelength
flaky GBIC/SFP vendors
small-time passive optical vendors
expensive equipment for testing (spectrum analyzer,
light sources, etc)
lack of operational expertise (get hit by a bus)
Circulators (same wave both ways)
Interleavers (half the light, double the waves)
CWDM light into DWDM channel (similar to above)
10GE LX4/LR multiplexing
2GE GWDM ~$1k
8GE CWDM ~$5k-10k
N*10GE DWDM ~N*$10k
prices include passive and active components,
per end, fdx over one pair
Prices an order of magnitude lower than commercial
systems from Cienna, Cisco.
List of vendors
Cloudy YAYA, Orient DONG,
[lots of names on slide, go read it yourself]
Questions? mail them!
Q: Martin, what do you about timing?
A: No need for timing, each channel is separate,
no timing needed to run this.
Q: mike hughes, linx; one thing to look at if you're
looking at GWDM/WWDM, or going bidir on one strand,
watch out for back reflections--running several channels
bidir would see itself reflected back, would declare
A: don't run two waves bidir on it--just don't do it,
it's not worth it, it's too ghetto.
Alerting prefix owners of hijacks in near-realtime
UCLA, joint work with a bunch of other names
[slides are at:
Three properties of a security solution
ability to see "bad" information
ability to distinguish between "good" and "bad" info
incentive to fix the problem
The PHAS (prefix hijack alert system) approach
use updates from existing BGP monitors (route views and
if false origination, send notification.
push complexity of detection to user
look at email registration to decide who is allowed to
don't filter out false vs real changes.
PHAS origin monitor
22.214.171.124/16, UCLA block
recommend multiple email addresses, including some that
are *not* on your blocks!
apply local rules before generating alarms
you shouldn't recieve duplicates of notifications
due to topological mesh-ness, it's difficult for a
hijacker to get all notifications for a block.
Evaluation: messages per AS
map prefixes to origin AS using routing table
most AS receive less than 100 messages per month
most less than 10
local filters can limit legitimate origin changes.
routeviews and RIPE RIS already collect data
alarm generation not dependent on
cooperation from other networks
monitoring or knowing correct origins
alarm authentication: single source
comprehensive study using archived data
developing near-realtime system
interested in receiving notifications
send email to:
covered prefix hijack
false last hop
PHAS: usenix security 2006.
Q: Danny McPherson--that's associated with origin AS,
and origin AS could be spoofed, does it look at
combination of prefix, origin, and next hop up?
A: they are doing it on origin AS and next hop,
they'll do some more thinking about that case.
Rick Wesson, Support Intelligence [hehe]
Understanding abuse, aggregate it, push it back to
operators, let them know what they're doing to other
[no slides, he does a live presentation of his tool]
How do I believe you?
realtime data visualization, Feb 8th, 2006
130 different data sources, 90% passive;
10,000 domain aggregated spam trap, very
evil SMTP that filters and bans IP for some time.
1.2million events per day aggregated, about 700,000
unique IPs for the global internet.
BGP peers, aggregate based on announcements made.
Put into tool so network operators can visualize
their prefixes, drill in, and see abuse each
hover over point, it shows the operator, IP address,
and what the problem was (spam, insecure web server, etc)
This shows problem areas that need to be addressed!
disseminate this information, help ISPs clean up their
Can also pass along information of abuse that has
happened to you.
If you have an AS, he can tell you what your AS has
been used for, abused for, owned, etc.
email him for more info...except he didn't list
his email info. ^_^;