A Study on Content-Based Reviewer Assignment in the Semantic Web and Computer Science Domains

Abstract

This paper underscores the pivotal role of high-quality paper reviews and their assignment to reviewers, delving into the intricate process of reviewer selection. Employing a comprehensive, multidisciplinary approach spanning computational science, information retrieval, and academic evaluation, our objective is to elevate the efficacy of the peer-review process. Our study involves a dataset in the Semantic Web and Computer Science domain, featuring 663 papers from 85 conferences and profiles of 524 reviewers. To assess the relevance of potential reviewers to scientific papers, we employ various similarity measures and representation strategies, including Jaccard similarity, dot product, and cosine similarity. Exploring different forms of representation, such as title-only, abstract-only, and a summary of the abstract generated with a Large Language Model-based tool, we utilize evaluation metrics like Mean Reciprocal Rank, Precision at $k$, and Mean Average Precision to validate the accuracy of reviewer recommendations. The culmination of our research offers valuable insights into effective reviewer selection strategies and optimal representation measures within the context of scientific paper evaluation. These findings contribute to the ongoing refinement of the peer-review process, enhancing its overall effectiveness.