The effect of discrete vs. continuous-valued ratings on reputation and ranking systems

¹ Département de Physique, Université de Fribourg - Chemin du Musée 3, CH-1700 Fribourg, Switzerland
² Institut Jean Nicod (CNRS) - 29 rue d'Ulm, F-75005 Paris, France, EU
³ CREATE-NET Research Consortium - via alla Cascata 56D, I-38123 Povo di Trento, Italy, EU

^a matus.medo@unifr.ch
^b joseph.wakeling@create-net.org

Received: 10 March 2010
Accepted: 9 August 2010

Abstract

When users rate objects, a sophisticated algorithm that takes into account ability or reputation may produce a fairer or more accurate aggregation of ratings than the straightforward arithmetic average. Recently a number of authors have proposed different co-determination algorithms where estimates of user and object reputation are refined iteratively together, permitting accurate measures of both to be derived directly from the rating data. However, simulations demonstrating these methods' efficacy assumed a continuum of rating values, consistent with typical physical modelling practice, whereas in most actual rating systems only a limited range of discrete values (such as a 5-star system) is employed. We perform a comparative test of several co-determination algorithms with different scales of discrete ratings and show that this seemingly minor modification in fact has a significant impact on algorithms' performance. Paradoxically, where rating resolution is low, increased noise in users' ratings may even improve the overall performance of the system.