Abstract
In today's Internet applications or sensor networks we often encounter large amounts of data spread over many physically distributed nodes. The sheer volume of the data and bandwidth constraints make it impractical to send all the data to one central node for query processing. Finding distributed icebergs - elements that may have low frequency at individual nodes but high aggregate frequency - is a problem that arises commonly in practice. In this paper we present a novel algorithm with two notable properties. First, its accuracy guarantee and communication cost are independent of the way in which element counts (for both icebergs and non-icebergs) are split amongst the nodes. Second, it works even when each distributed data set is a stream (i.e., one pass data access only). Our algorithm builds upon sketches constructed for the estimation of the second frequency moment (F2) of data streams. The intuition of our idea is that when there are global icebergs in the union of these data streams the F2 of the union becomes very large. This quantity can be estimated due to the summable nature of F2 sketches. Our key innovation here is to establish tight theoretical guarantees of our algorithm, under certain reasonable assumptions, using an interesting combination of convex ordering theory and large deviation techniques.
| Original language | English (US) |
|---|---|
| Title of host publication | Proceedings - International Conference on Data Engineering |
| Pages | 557-568 |
| Number of pages | 12 |
| DOIs | |
| State | Published - 2010 |
| Event | 26th IEEE International Conference on Data Engineering, ICDE 2010 - Long Beach, CA, United States Duration: Mar 1 2010 → Mar 6 2010 |
Other
| Other | 26th IEEE International Conference on Data Engineering, ICDE 2010 |
|---|---|
| Country | United States |
| City | Long Beach, CA |
| Period | 3/1/10 → 3/6/10 |
Fingerprint
ASJC Scopus subject areas
- Information Systems
- Signal Processing
- Software
Cite this
Global iceberg detection over distributed data streams. / Zhao, Haiquan; Lall, Ashwin; Ogihara, Mitsunori; Xu, Jun.
Proceedings - International Conference on Data Engineering. 2010. p. 557-568 5447825.Research output: Chapter in Book/Report/Conference proceeding › Conference contribution
}
TY - GEN
T1 - Global iceberg detection over distributed data streams
AU - Zhao, Haiquan
AU - Lall, Ashwin
AU - Ogihara, Mitsunori
AU - Xu, Jun
PY - 2010
Y1 - 2010
N2 - In today's Internet applications or sensor networks we often encounter large amounts of data spread over many physically distributed nodes. The sheer volume of the data and bandwidth constraints make it impractical to send all the data to one central node for query processing. Finding distributed icebergs - elements that may have low frequency at individual nodes but high aggregate frequency - is a problem that arises commonly in practice. In this paper we present a novel algorithm with two notable properties. First, its accuracy guarantee and communication cost are independent of the way in which element counts (for both icebergs and non-icebergs) are split amongst the nodes. Second, it works even when each distributed data set is a stream (i.e., one pass data access only). Our algorithm builds upon sketches constructed for the estimation of the second frequency moment (F2) of data streams. The intuition of our idea is that when there are global icebergs in the union of these data streams the F2 of the union becomes very large. This quantity can be estimated due to the summable nature of F2 sketches. Our key innovation here is to establish tight theoretical guarantees of our algorithm, under certain reasonable assumptions, using an interesting combination of convex ordering theory and large deviation techniques.
AB - In today's Internet applications or sensor networks we often encounter large amounts of data spread over many physically distributed nodes. The sheer volume of the data and bandwidth constraints make it impractical to send all the data to one central node for query processing. Finding distributed icebergs - elements that may have low frequency at individual nodes but high aggregate frequency - is a problem that arises commonly in practice. In this paper we present a novel algorithm with two notable properties. First, its accuracy guarantee and communication cost are independent of the way in which element counts (for both icebergs and non-icebergs) are split amongst the nodes. Second, it works even when each distributed data set is a stream (i.e., one pass data access only). Our algorithm builds upon sketches constructed for the estimation of the second frequency moment (F2) of data streams. The intuition of our idea is that when there are global icebergs in the union of these data streams the F2 of the union becomes very large. This quantity can be estimated due to the summable nature of F2 sketches. Our key innovation here is to establish tight theoretical guarantees of our algorithm, under certain reasonable assumptions, using an interesting combination of convex ordering theory and large deviation techniques.
UR - http://www.scopus.com/inward/record.url?scp=77952758693&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=77952758693&partnerID=8YFLogxK
U2 - 10.1109/ICDE.2010.5447825
DO - 10.1109/ICDE.2010.5447825
M3 - Conference contribution
SN - 9781424454440
SP - 557
EP - 568
BT - Proceedings - International Conference on Data Engineering
ER -