With the proliferation of multimedia data and ever-growing requests for multimedia applications, new challenges are emerged for efficient and effective managing and accessing large audio-visual collections. In this paper, we present a novel framework for video event detection, which plays an essential role in high-level video indexing and retrieval. Especially, since temporal information in a video sequence is critical in conveying video content, a hierarchical temporal association mining approach is developed to systematically capture the characteristic temporal patterns with respect to the events of interest. In this process, the unique challenges caused by the loose video structure and skewed data distribution issues are effectively tackled. In addition, an adaptive mechanism is proposed to determine the essential thresholds which are generally defined manually in the traditional association rule mining (ARM) approach. This framework thus largely relaxes the dependence on the domain knowledge and contributes to the ultimate goal of automatic video content analysis.