The choice of a suitable backhaul constitutes one of the main performance bottlenecks in the emerging femtocell networks. In this paper, we study the impact of adopting a heterogenous backhaul (i.e., wired or over-the-air) with realistic quality-of-service requirements on coherent coordinated multipoint (CoMP) transmission in the downlink of femtocell networks. We formulate a cooperative game with continuum among the femtocell access points (FAPs) for performing CoMP in order to maximize the downlink rate while accounting for the constraints on the heterogeneous backhaul. In this respect, we propose a distributed algorithm that enables the FAPs to jointly decide on their cooperative partners as well as the choice of a backhaul strategy. In this respect, the proposed algorithm jointly addresses the problem of coalition formation as well as the optimization of the tradeoff between OTA and wired backhaul transmission modes, each of which is limited by a different factor such as delay or spectrum resources availability. We show that the proposed algorithm converges to a stable partition which constitutes the continuum core of the studied cooperative game. Simulation results show that our proposed scheme yields interesting gains in terms of the average downlink rate per FAP, reaching up to 26% relative to the classical of non-cooperative transmissions.