In recent years, E-commerce has become one of the fastest growing industry and constantly provided better services for the customers. Such incredible advance was partially thanks to the mature development of the semantic web technologies with better content support for the search engines and recommender systems. Moreover, with rich embedded information, hypervideo has turned out to be a promising way of online shopping. In order to facilitate its development, GoodRelations ontology has been developed to make the video more comprehensive for both the users and machines. However, due to the vastness and diversity of the internet, the existing methodologies could not reach a consensus on the information annotated by the participants. Therefore, in this study, we propose a semantic video annotation framework using consensus quality of the collaborative process and GoodRelations ontology as the domain knowledge. This approach has been demonstrated to enhance the quality of the annotated information, but still maintain the flexibility for the vendors to sell their products interactively on the videos. As a result, this study provides a robust and reliable video annotation platform to be used in the E-commerce industry.