In this research, we propose an integrated and interactive framework to manage and retrieve large scale video archives. The video data are modeled by a hierarchical learning mechanism called HMMM (Hierarchical Markov Model Mediator) and indexed by an innovative semantic video database clustering strategy. The cumulated user feedbacks are reused to update the affinity relationships of the video objects as well as their initial state probabilities. Correspondingly, both the high level semantics and user perceptions are employed in the video clustering strategy. The clustered video database is capable of providing appealing multimedia experience to the users because the modeled multimedia database system can learn the user's preferences and interests interactively.