Ranking with Ties Based on Noisy Performance Data

We consider the problem of ranking a set of objects based on theirperformance when the measurement of said performance is subject to noise. Inthis scenario, the performance is measured repeatedly, resulting in a range ofmeasurements for each object. If the ranges of two objects do not overlap, thenwe consider one object as 'better' than the other, and we expect it to receivea higher rank; if, however, the ranges overlap, then the objects areincomparable, and we wish them to be assigned the same rank. Unfortunately, theincomparability relation of ranges is in general not transitive; as aconsequence, in general the two requirements cannot be satisfiedsimultaneously, i.e., it is not possible to guarantee both distinct ranks forobjects with separated ranges, and same rank for objects with overlappingranges. This conflict leads to more than one reasonable way to rank a set ofobjects. In this paper, we explore the ambiguities that arise when ranking withties, and define a set of reasonable rankings, which we call partial rankings.We develop and analyse three different methodologies to compute a partialranking. Finally, we show how performance differences among objects can beinvestigated with the help of partial ranking.