Active learning for streaming data in a contextual bandit framework

Contextual bandit algorithms have been shown to be effective in solving sequential decision making problems under uncertainty. A common assumption in the literature is that the realized (ground truth) reward is observed by the learner at no cost, which, however, is not realistic in many practical scenarios. When observing the ground truth reward is costly, a key challenge is how to judiciously acquire the ground truth by assessing the benefits and costs in order to balance learning efficiency and learning cost. In this paper, we design a novel contextual bandit-based learning algorithm and endow it with the active learning capability. In addition to sending a query to an annotator for the ground truth, prior information about the ground truth learned by the learner is sent together, thereby reducing the query cost. We prove that the learning regret of the proposed algorithm achieves the same order as that of conventional contextual bandit algorithms in cost-free scenarios, implying that, surprisingly, cost due to acquiring the ground truth does not increase the learning regret in the long-run, where the prior information about the ground truth plays a critical role.