A data streaming algorithm for estimating entropies of OD flows

Haiquan Zhao, Oliver Spatscheck, Ashwin Lall, Jia Wang, Mitsunori Ogihara, Jun Xu

Research output: Chapter in Book/Report/Conference proceedingConference contribution

29 Citations (Scopus)

Abstract

Entropy has recently gained considerable significance as an important metric for network measurement. Previous research has shown its utility in clustering traffic and detecting traffic anomalies. While measuring the entropy of the traffic observed at a single point has already been studied, an interesting open problem is to measure the entropy of the traffic between every origin-destination pair. In this paper, we propose the first solution to this challenging problem. Our sketch builds upon and extends the Lp sketch of Indyk with significant additional innovations. We present calculations showing that our data streaming algorithm is feasible for high link speeds using commodity CPU/memory at a reasonable cost. Our algorithm is shown to be very accurate in practice via simulations, using traffic traces collected at a tier-1 ISP backbone link.

Original languageEnglish (US)
Title of host publicationProceedings of the ACM SIGCOMM Internet Measurement Conference, IMC
Pages279-290
Number of pages12
DOIs
StatePublished - 2007
Externally publishedYes
EventIMC'07: 2007 7th ACM SIGCOMM Internet Measurement Conference - San Diego, CA, United States
Duration: Oct 24 2007Oct 26 2007

Other

OtherIMC'07: 2007 7th ACM SIGCOMM Internet Measurement Conference
CountryUnited States
CitySan Diego, CA
Period10/24/0710/26/07

Fingerprint

Entropy
Program processors
Innovation
Data storage equipment
Costs

Keywords

  • Data streaming
  • Entropy estimation
  • Network measurement
  • Stable distributions
  • Traffic matrix

ASJC Scopus subject areas

  • Engineering(all)

Cite this

Zhao, H., Spatscheck, O., Lall, A., Wang, J., Ogihara, M., & Xu, J. (2007). A data streaming algorithm for estimating entropies of OD flows. In Proceedings of the ACM SIGCOMM Internet Measurement Conference, IMC (pp. 279-290) https://doi.org/10.1145/1298306.1298345

A data streaming algorithm for estimating entropies of OD flows. / Zhao, Haiquan; Spatscheck, Oliver; Lall, Ashwin; Wang, Jia; Ogihara, Mitsunori; Xu, Jun.

Proceedings of the ACM SIGCOMM Internet Measurement Conference, IMC. 2007. p. 279-290.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Zhao, H, Spatscheck, O, Lall, A, Wang, J, Ogihara, M & Xu, J 2007, A data streaming algorithm for estimating entropies of OD flows. in Proceedings of the ACM SIGCOMM Internet Measurement Conference, IMC. pp. 279-290, IMC'07: 2007 7th ACM SIGCOMM Internet Measurement Conference, San Diego, CA, United States, 10/24/07. https://doi.org/10.1145/1298306.1298345
Zhao H, Spatscheck O, Lall A, Wang J, Ogihara M, Xu J. A data streaming algorithm for estimating entropies of OD flows. In Proceedings of the ACM SIGCOMM Internet Measurement Conference, IMC. 2007. p. 279-290 https://doi.org/10.1145/1298306.1298345
Zhao, Haiquan ; Spatscheck, Oliver ; Lall, Ashwin ; Wang, Jia ; Ogihara, Mitsunori ; Xu, Jun. / A data streaming algorithm for estimating entropies of OD flows. Proceedings of the ACM SIGCOMM Internet Measurement Conference, IMC. 2007. pp. 279-290
@inproceedings{455860e71bb0474b9ba4eeb913c45191,
title = "A data streaming algorithm for estimating entropies of OD flows",
abstract = "Entropy has recently gained considerable significance as an important metric for network measurement. Previous research has shown its utility in clustering traffic and detecting traffic anomalies. While measuring the entropy of the traffic observed at a single point has already been studied, an interesting open problem is to measure the entropy of the traffic between every origin-destination pair. In this paper, we propose the first solution to this challenging problem. Our sketch builds upon and extends the Lp sketch of Indyk with significant additional innovations. We present calculations showing that our data streaming algorithm is feasible for high link speeds using commodity CPU/memory at a reasonable cost. Our algorithm is shown to be very accurate in practice via simulations, using traffic traces collected at a tier-1 ISP backbone link.",
keywords = "Data streaming, Entropy estimation, Network measurement, Stable distributions, Traffic matrix",
author = "Haiquan Zhao and Oliver Spatscheck and Ashwin Lall and Jia Wang and Mitsunori Ogihara and Jun Xu",
year = "2007",
doi = "10.1145/1298306.1298345",
language = "English (US)",
isbn = "9781595939081",
pages = "279--290",
booktitle = "Proceedings of the ACM SIGCOMM Internet Measurement Conference, IMC",

}

TY - GEN

T1 - A data streaming algorithm for estimating entropies of OD flows

AU - Zhao, Haiquan

AU - Spatscheck, Oliver

AU - Lall, Ashwin

AU - Wang, Jia

AU - Ogihara, Mitsunori

AU - Xu, Jun

PY - 2007

Y1 - 2007

N2 - Entropy has recently gained considerable significance as an important metric for network measurement. Previous research has shown its utility in clustering traffic and detecting traffic anomalies. While measuring the entropy of the traffic observed at a single point has already been studied, an interesting open problem is to measure the entropy of the traffic between every origin-destination pair. In this paper, we propose the first solution to this challenging problem. Our sketch builds upon and extends the Lp sketch of Indyk with significant additional innovations. We present calculations showing that our data streaming algorithm is feasible for high link speeds using commodity CPU/memory at a reasonable cost. Our algorithm is shown to be very accurate in practice via simulations, using traffic traces collected at a tier-1 ISP backbone link.

AB - Entropy has recently gained considerable significance as an important metric for network measurement. Previous research has shown its utility in clustering traffic and detecting traffic anomalies. While measuring the entropy of the traffic observed at a single point has already been studied, an interesting open problem is to measure the entropy of the traffic between every origin-destination pair. In this paper, we propose the first solution to this challenging problem. Our sketch builds upon and extends the Lp sketch of Indyk with significant additional innovations. We present calculations showing that our data streaming algorithm is feasible for high link speeds using commodity CPU/memory at a reasonable cost. Our algorithm is shown to be very accurate in practice via simulations, using traffic traces collected at a tier-1 ISP backbone link.

KW - Data streaming

KW - Entropy estimation

KW - Network measurement

KW - Stable distributions

KW - Traffic matrix

UR - http://www.scopus.com/inward/record.url?scp=42149164252&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=42149164252&partnerID=8YFLogxK

U2 - 10.1145/1298306.1298345

DO - 10.1145/1298306.1298345

M3 - Conference contribution

SN - 9781595939081

SP - 279

EP - 290

BT - Proceedings of the ACM SIGCOMM Internet Measurement Conference, IMC

ER -