Data streaming algorithms for estimating entropy of network traffic

Ashwin Lall, Vyas Sekar, Mitsunori Ogihara, Jun Xu, Hui Zhang

Research output: Chapter in Book/Report/Conference proceedingConference contribution

93 Scopus citations

Abstract

Using entropy of traffic distributions has been shown to aid a wide variety of network monitoring applications such as anomaly detection, clustering to reveal interesting patterns, and traffic classification. However, realizing this potential benefit in practice requires accurate algorithms that can operate on high-speed links, with low CPU and memory requirements. In this paper, we investigate the problem of estimating the entropy in a streaming computation model. We give lower bounds for this problem, showing that neither approximation nor randomization alone will let us compute the entropy efficiently. We present two algorithms for randomly approximating the entropy in a time and space efficient manner, applicable for use on very high speed (greater than OC-48) links. The first algorithm for entropy estimation is inspired by the structural similarity with the seminal work of Alon et al. for estimating frequency moments, and we provide strong theoretical guarantees on the error and resource usage. Our second algorithm utilizes the observation that the performance of the streaming algorithm can be enhanced by separating the high-frequency items (or elephants) from the low-frequency items (or mice). We evaluate our algorithms on traffic traces from different deployment scenarios.

Original languageEnglish (US)
Title of host publicationSIGMETRICS 2006/Performance 2006 - Joint International Conference on Measurement and Modeling of Computer Systems, Proceedings
Pages145-156
Number of pages12
Edition1
DOIs
StatePublished - Jun 1 2006
Externally publishedYes
EventSIGMETRICS 2006/Performance 2006 - Joint International Conference on Measurement and Modeling of Computer Systems - Saint Malo, France
Duration: Jun 26 2006Jun 30 2006

Publication series

NamePerformance Evaluation Review
Number1
Volume34
ISSN (Print)0163-5999
ISSN (Electronic)0163-5999

Other

OtherSIGMETRICS 2006/Performance 2006 - Joint International Conference on Measurement and Modeling of Computer Systems
CountryFrance
CitySaint Malo
Period6/26/066/30/06

    Fingerprint

Keywords

  • Data streaming
  • Traffic analysis

ASJC Scopus subject areas

  • Software
  • Hardware and Architecture
  • Computer Networks and Communications

Cite this

Lall, A., Sekar, V., Ogihara, M., Xu, J., & Zhang, H. (2006). Data streaming algorithms for estimating entropy of network traffic. In SIGMETRICS 2006/Performance 2006 - Joint International Conference on Measurement and Modeling of Computer Systems, Proceedings (1 ed., pp. 145-156). (Performance Evaluation Review; Vol. 34, No. 1). https://doi.org/10.1145/1140103.1140295