An efficient algorithm for measuring medium- to large-sized flows in network traffic

Ashwin Lall, Mitsunori Ogihara, Jun Xu

Research output: Chapter in Book/Report/Conference proceedingConference contribution

4 Citations (Scopus)

Abstract

It has been well recognized that identifying very large flows (i.e., elephants) in a network traffic stream is important for a variety of network applications ranging from traffic engineering to anomaly detection. However, we found that many of these applications have an increasing need to monitor not only the few largest flows (say top 20), but also all of the medium-sized flows (say top 20,000). Unfortunately, existing techniques for identifying elephant flows at high link speeds are not suitable and cannot be trivially extended for identifying the medium-sized flows. In this work, we propose a hybrid SRAM/DRAM algorithm for monitoring all elephant and medium-sized flows with strong accuracy guarantees. We employ a synopsis data structure (sketch) in SRAM to filter out small flows and preferentially sample medium and large flows to a flow table in DRAM. Our key contribution is to show how to maximize the use of SRAM and DRAM available to us by using a SRAM/DRAM hybrid data structure that can achieve more than an order of magnitude higher SRAM efficiency than previous methods. We design a quantization scheme that allows our algorithm to "read just enough" from the sketch at SRAM speed, without sacrificing much estimation accuracy. We provide analytical guarantees on the accuracy of the estimation and validate these by means of trace-driven evaluation using real-world packet traces.

Original languageEnglish (US)
Title of host publicationProceedings - IEEE INFOCOM
Pages2711-2715
Number of pages5
DOIs
StatePublished - 2009
Event28th Conference on Computer Communications, IEEE INFOCOM 2009 - Rio de Janeiro, Brazil
Duration: Apr 19 2009Apr 25 2009

Other

Other28th Conference on Computer Communications, IEEE INFOCOM 2009
CountryBrazil
CityRio de Janeiro
Period4/19/094/25/09

Fingerprint

Static random access storage
Dynamic random access storage
Data structures
Monitoring

ASJC Scopus subject areas

  • Computer Science(all)
  • Electrical and Electronic Engineering

Cite this

An efficient algorithm for measuring medium- to large-sized flows in network traffic. / Lall, Ashwin; Ogihara, Mitsunori; Xu, Jun.

Proceedings - IEEE INFOCOM. 2009. p. 2711-2715 5062217.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Lall, A, Ogihara, M & Xu, J 2009, An efficient algorithm for measuring medium- to large-sized flows in network traffic. in Proceedings - IEEE INFOCOM., 5062217, pp. 2711-2715, 28th Conference on Computer Communications, IEEE INFOCOM 2009, Rio de Janeiro, Brazil, 4/19/09. https://doi.org/10.1109/INFCOM.2009.5062217
Lall, Ashwin ; Ogihara, Mitsunori ; Xu, Jun. / An efficient algorithm for measuring medium- to large-sized flows in network traffic. Proceedings - IEEE INFOCOM. 2009. pp. 2711-2715
@inproceedings{dbef659a8e484ad6840a2094af658938,
title = "An efficient algorithm for measuring medium- to large-sized flows in network traffic",
abstract = "It has been well recognized that identifying very large flows (i.e., elephants) in a network traffic stream is important for a variety of network applications ranging from traffic engineering to anomaly detection. However, we found that many of these applications have an increasing need to monitor not only the few largest flows (say top 20), but also all of the medium-sized flows (say top 20,000). Unfortunately, existing techniques for identifying elephant flows at high link speeds are not suitable and cannot be trivially extended for identifying the medium-sized flows. In this work, we propose a hybrid SRAM/DRAM algorithm for monitoring all elephant and medium-sized flows with strong accuracy guarantees. We employ a synopsis data structure (sketch) in SRAM to filter out small flows and preferentially sample medium and large flows to a flow table in DRAM. Our key contribution is to show how to maximize the use of SRAM and DRAM available to us by using a SRAM/DRAM hybrid data structure that can achieve more than an order of magnitude higher SRAM efficiency than previous methods. We design a quantization scheme that allows our algorithm to {"}read just enough{"} from the sketch at SRAM speed, without sacrificing much estimation accuracy. We provide analytical guarantees on the accuracy of the estimation and validate these by means of trace-driven evaluation using real-world packet traces.",
author = "Ashwin Lall and Mitsunori Ogihara and Jun Xu",
year = "2009",
doi = "10.1109/INFCOM.2009.5062217",
language = "English (US)",
isbn = "9781424435135",
pages = "2711--2715",
booktitle = "Proceedings - IEEE INFOCOM",

}

TY - GEN

T1 - An efficient algorithm for measuring medium- to large-sized flows in network traffic

AU - Lall, Ashwin

AU - Ogihara, Mitsunori

AU - Xu, Jun

PY - 2009

Y1 - 2009

N2 - It has been well recognized that identifying very large flows (i.e., elephants) in a network traffic stream is important for a variety of network applications ranging from traffic engineering to anomaly detection. However, we found that many of these applications have an increasing need to monitor not only the few largest flows (say top 20), but also all of the medium-sized flows (say top 20,000). Unfortunately, existing techniques for identifying elephant flows at high link speeds are not suitable and cannot be trivially extended for identifying the medium-sized flows. In this work, we propose a hybrid SRAM/DRAM algorithm for monitoring all elephant and medium-sized flows with strong accuracy guarantees. We employ a synopsis data structure (sketch) in SRAM to filter out small flows and preferentially sample medium and large flows to a flow table in DRAM. Our key contribution is to show how to maximize the use of SRAM and DRAM available to us by using a SRAM/DRAM hybrid data structure that can achieve more than an order of magnitude higher SRAM efficiency than previous methods. We design a quantization scheme that allows our algorithm to "read just enough" from the sketch at SRAM speed, without sacrificing much estimation accuracy. We provide analytical guarantees on the accuracy of the estimation and validate these by means of trace-driven evaluation using real-world packet traces.

AB - It has been well recognized that identifying very large flows (i.e., elephants) in a network traffic stream is important for a variety of network applications ranging from traffic engineering to anomaly detection. However, we found that many of these applications have an increasing need to monitor not only the few largest flows (say top 20), but also all of the medium-sized flows (say top 20,000). Unfortunately, existing techniques for identifying elephant flows at high link speeds are not suitable and cannot be trivially extended for identifying the medium-sized flows. In this work, we propose a hybrid SRAM/DRAM algorithm for monitoring all elephant and medium-sized flows with strong accuracy guarantees. We employ a synopsis data structure (sketch) in SRAM to filter out small flows and preferentially sample medium and large flows to a flow table in DRAM. Our key contribution is to show how to maximize the use of SRAM and DRAM available to us by using a SRAM/DRAM hybrid data structure that can achieve more than an order of magnitude higher SRAM efficiency than previous methods. We design a quantization scheme that allows our algorithm to "read just enough" from the sketch at SRAM speed, without sacrificing much estimation accuracy. We provide analytical guarantees on the accuracy of the estimation and validate these by means of trace-driven evaluation using real-world packet traces.

UR - http://www.scopus.com/inward/record.url?scp=70349675311&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=70349675311&partnerID=8YFLogxK

U2 - 10.1109/INFCOM.2009.5062217

DO - 10.1109/INFCOM.2009.5062217

M3 - Conference contribution

SN - 9781424435135

SP - 2711

EP - 2715

BT - Proceedings - IEEE INFOCOM

ER -