The dream of pervasive multimedia retrieval and reuse will not be realized without incorporating semantics in the multimedia database. As video data is penetrating many information systems, the need for database support for video data evolves. Hence, we propose an innovative database modeling mechanism called Hierarchical Markov Model Mediator (HMMM) which integrates lowlevel features, semantic concepts, and high-level user perceptions for modeling and indexing multiple-level video objects to facilitate temporal pattern retrieval. Different from the existing database modeling methods, our approach carries a stochastic and dynamic process in both search and similarity calculation. In the retrieval of semantic event patterns, HMMM always tries to traverse the right path and therefore it can assist in retrieving more accurate patterns quickly with lower computational costs. Moreover, HMMM supports feedbacks and learning strategies, which can proficiently assure the continuous improvements of the overall performance.