Consistent asymetric latency on monitoring?

[Rick Ernst]

Although the implementation is Cisco-specific, this feels more appropriate
for NANOG.

We've started rolling out a state-wide monitoring system based on Cisco's
"IP SLA" feature set.  Out of 5 sites deployed so far (different locations,
different providers), we are consistently seeing one-way latency mirror the
opposite direction. As source-destination latency goes up,
destination-source latency goes down and vice versa.

Myself and the monitoring team have ripped apart the OIDs, IP SLA
configuration, and monitoring system.  We've also built an ad-hoc system to
compare the results.  It's still consistent behavior.  It's not a true
mirror; there is definitely variation between the data collection, but at
the 10,000 foot level, there is an obvious and consistent mirror to the
data.

The network topology is independant service providers all providing backhaul
to a local ethernet exchange.

Has anybody seen this type of behavior? We are solidly convinced that we are
using the proper OIDs and making the proper transformations of the data.
The two remaining causes appear to be either "natural behavior of the links"
and/or "artifact in the IP SLA mechanism".

Any ideas?


Thanks!