Automated Ensemble for Deep Learning Inference on Edge Computing Platforms

Yang Bai, Lixing Chen, Mohamed Abdel-Mottaleb, Jie Xu

Research output: Contribution to journalArticlepeer-review


Advances in deep learning (DL) have triggered an explosion of mobile intelligence, posing a soaring demand for computing resources that cannot be satisfied by mobile devices. In this paper, we employ Edge Computing to deliver better DL inference services to end-users. The key is to leverage deep neural network (DNN) ensemble techniques that provide state-of-the-art performance for many machine learning applications in terms of inference accuracy and robustness. Compared to end-devices, the edge computing platform is endowed with more powerful computing resources, making it feasible to implement DNN ensembles for DL inferences. However, due to the constrained computing capacity of edge servers and the possible service response deadline, an edge server can only use a limited number of DNNs to construct DNN ensembles. This poses a unique problem, namely DNN ensemble selection, for identifying the best-fit DNN ensembles. We propose a novel algorithm called Automated DNN Ensemble (AES) algorithm to solve this problem. Because DNNs exhibit performance variations over different distributions of input data, AES adaptively determines a DNN ensemble according to the features of admitted inference tasks. AES is an online learning algorithm that learns DNNs’ in-use performance over time. An ensemble selection rule is further designed as a subroutine of AES to recruit members into the DNN ensemble based on the accuracy and diversity of DNNs. In particular, we theoretically prove that AES can achieve asymptotic optimality. We carry out experiments on real-world datasets. The results show that using the DNN ensemble technique on edge computing platforms dramatically improves the DL inference quality, and AES outperforms other benchmark schemes.

Original languageEnglish (US)
JournalIEEE Internet of Things Journal
StateAccepted/In press - 2021


  • Deep learning
  • deep neural network ensemble.
  • Edge computing
  • Edge computing
  • Inference algorithms
  • Mobile handsets
  • multi-armed bandit
  • Performance evaluation
  • Servers
  • Task analysis

ASJC Scopus subject areas

  • Signal Processing
  • Information Systems
  • Hardware and Architecture
  • Computer Science Applications
  • Computer Networks and Communications


Dive into the research topics of 'Automated Ensemble for Deep Learning Inference on Edge Computing Platforms'. Together they form a unique fingerprint.

Cite this