Semantic similarity through hierarchical abstraction of knowledge

Abstract

Identifying semantic similarity between two texts has many applications in NLP including information extraction and retrieval, word sense disambigua- tion, text summarization and type classi cation. Similarity between texts is commonly determined using a taxonomy based approach, but the limited scalability of existing taxonomies has led recent research to use Wikipedia's encyclopaedic knowledge base to nd similarity or relatedness. In this the- sis, we propose Hierarchical Semantic Analysis, a method which represents semantics of a text in high dimensional space of Wikipedia concepts and category hierarchies. We represent the meaning of any text excerpt as a weighed vector of Wikipedia-based resources. To evaluate the similarity of texts in this space, we compare the corresponding vectors using conventional metrics (e.g. cosine). Compared with the previous state of the art, use of Hierarchical Semantic Analysis(HSA) results in substantial improvements in correlation of computed similarity scores with human judgements from r= .873 to 0.901 for short sentence pairs and from r= .72 to 0.863 for paragraph pairs.