Publication announcement:

Multi-layer network monitoring and analysis

James Hall

Technical report UCAM-CL-TR-571, University of Cambridge,
Computer Laboratory, July 2003, 230 pages.

This document is now available at

http://www.cl.cam.ac.uk/TechReports/UCAM-CL-TR-571.pdf

Abstract:

Passive network monitoring offers the possibility of gathering a wealth
of data about the traffic traversing the network and the communicating
processes generating that traffic. Significant advantages include the
non-intrusive nature of data capture and the range and diversity of the
traffic and driving applications which may be observed. Conversely there
are also associated practical difficulties which have restricted the
usefulness of the technique: increasing network bandwidths can challenge
the capacity of monitors to keep pace with passing traffic without data
loss, and the bulk of data recorded may become unmanageable.

Much research based upon passive monitoring has in consequence been
limited to that using a sub-set of the data potentially available,
typically TCP/IP packet headers gathered using Tcpdump or similar
monitoring tools. The bulk of data collected is thereby minimised, and
with the possible exception of packet filtering, the monitor's available
processing power is available for the task of collection and storage. As
the data available for analysis is drawn from only a small section of
the network protocol stack, detailed study is largely confined to the
associated functionality and dynamics in isolation from activity at
other levels. Such lack of context severely restricts examination of the
interaction between protocols which may in turn lead to inaccurate or
erroneous conclusions.

The work described in this report attempts to address some of these
limitations. A new passive monitoring architecture -- Nprobe -- is
presented, based upon 'off the shelf' components and which, by using
clusters of probes, is scalable to keep pace with current high bandwidth
networks without data loss. Monitored packets are fully captured, but
are subject to the minimum processing in real time needed to identify
and associate data of interest across the target set of protocols. Only
this data is extracted and stored. The data reduction ratio thus
achieved allows examination of a wider range of encapsulated protocols
without straining the probe's storage capacity.

Full analysis of the data harvested from the network is performed
off-line. The activity of interest within each protocol is examined and
is integrated across the range of protocols, allowing their interaction
to be studied. The activity at higher levels informs study of the lower
levels, and that at lower levels infers detail of the higher. A
technique for dynamically modelling TCP connections is presented, which,
by using data from both the transport and higher levels of the protocol
stack, differentiates between the effects of network and end-process
activity.

The balance of the report presents a study of Web traffic using Nprobe.
Data collected from the IP, TCP, HTTP and HTML levels of the stack is
integrated to identify the patterns of network activity involved in
downloading whole Web pages: by using the links contained in HTML
documents observed by the monitor, together with data extracted from the
HTML headers of downloaded contained objects, the set of TCP connections
used, and the way in which browsers use them, are studied as a whole. An
analysis of the degree and distribution of delay is presented and
contributes to the understanding of performance as perceived by the
user. The effects of packet loss on whole page download times are
examined, particularly those losses occurring early in the lifetime of
connections before reliable estimations of round trip times are
established. The implications of such early packet losses for pages
downloads using persistent connections are also examined by simulations
using the detailed data available.

--
University of Cambridge, Computer Laboratory,
Technical Reports (ISSN 1476-2986)
http://www.cl.cam.ac.uk/TechReports/