„Good judgment comes from experience, and experience… well that comes from poor judgment”

Bernard Baruch

„Pairwise comparison generally refers to any process of comparing entities in pairs to judge which of each entity is preferred, or has a greater amount of some quantitative property” (Wikipedia). In my opinion, the Pairwise Comparisons Method is an essential part of Decision Theory. Moreover, it is a part of the theoretical basis of decision support systems. Although the pairwise comparisons method (PCM) has a very long story, there is still some room for improvement.

Comparing alternatives in pairs is a very well known technique of ranking creation. The answer to how reliable and trustworthy ranking is depends on the inconsistency of the data from which it was created. There are many indices used for determining the level of inconsistency among compared alternatives. Unfortunately, most of them assume that the set of comparisons is complete, i.e. every single alternative is compared to each other. This is not true and the ranking must sometimes be made based on incomplete data. In order to fill this gap, this work aims to adapt the selected twelve existing inconsistency indices for the purpose of analyzing incomplete data sets. The modified indices are subjected to Monte Carlo experiments. Those of them that achieved the best results in the experiments carried out are recommended for use in practice.

Arxiv preprint of "Kułakowski & Talaga, Inconsistency indices for incomplete pairwise comparisons matrices, 2019"

• Montecarlo experiment Montecarlo - code listing

• The complete sources of the experiment can be get here. If you want to reuse the code please cite the paper mentioned above.

Public lectures at AGH University of Science and Technology, Cracow, Poland (22, 29 October 2014)

• Lecture 1: Heuristic Rating Estimation 
- Existence of Solution
• Lecture 2: HRE - Geometric Approach
• Lecture 3: Hierarchical Bid Assessment (HBA) model 
- pairwise comparisons approach

The pairwise comparisons method is a convenient tool used when the relative order among different concepts (alternatives) needs to be determined. One popular implementation of the method is based on solving an eigenvalue problem for the pairwise comparisons matrix. In such cases the ranking result for the principal eigenvector of the pairwise comparisons matrix is adopted, while the eigenvalue is used to determine the index of inconsistency. A lot of research has been devoted to the critical analysis of the eigenvalue based approach. One of them is the work of Bana e Costa and Vansnick (2008). In their work, the authors define the conditions of order preservation (COP) and show that even for sufficiently consistent pairwise comparisons matrices, this condition cannot be met. The presented work defines more precise criteria for determining when the COP is met. To formulate the criteria, an error factor is used describing how far the input to the ranking procedure is from the ranking result. The relationship between the Saaty consistency index and COP is also discussed.

The pairwise comparisons method can be used when the relative order of preferences among different concepts (alternatives) needs to be determined. There are several popular imple- mentations of this method, including the Eigenvector Method, the Least Squares Method, the Chi Squares Method and others. Each of the above methods comes with one or more inconsistency in- dices that help to decide whether the consistency of input guarantees obtaining a reliable output, thus taking the optimal decision. This article explores the relationship between inconsistency of input and error of output. An error describes to what extent the obtained results correspond to the single expert’s assessments. On the basis of the inconsistency and the error, two properties of the weight deriving procedure are formulated. These properties are proven for eigenvector method and Koczkodaj’s inconsistency index. Several estimates using Koczkodaj’s inconsistency index for a principal eigenvalue, Saaty’s inconsistency index and the Condition of Order Preservation are also provided.

This paper presents a concurrent algorithm for computing a consistent approximation to a generalized pairwise comparisons ma- trix (i.e. it is assumed that the reciprocity property is not required). Like its sequential counterpart, it is based on the iterative strategy “find the worst case and fix it”. The conducted experiments confirmed that a significant increase in speed between the sequential and concurrent ap- proach is achieved. Our results may be particularly important for the large decision support systems where the number of pairs considered is large and the sequential approach may not be fast enough.

Pairwise comparisons (PC) is a well-known method for modeling the subjective preferences of a decision maker. The method is very often used in the models of voting systems, social choice theory, decision techniques (such as AHP - Analytic Hierarchy Process) or multi-agent AI systems. In this approach, a set of paired comparisons is transformed into one overall ranking of alternatives. Very often, only the results of individual comparisons are given, whilst the weights (indicators of significance) of the alternatives need to be computed. According to Heuristic Rating Estimation (HRE), the new approach discussed in the article, besides the results of comparisons, the weights of some alternatives can also be a priori known. Although HRE uses a similar method to the popular AHP technique to compute the weights of individual alternatives, the solution obtained is not always positive and real. This article tries to answer the question of when such a correct solution exists. Hence, the sufficient condition for the existence of a positive and real solution in the HRE approach is formulated and proven. The influence of inconsistency in the paired comparisons set for the existence of a solution is also discussed.

Unfortunately, in the original publication, the conditions Q1, Q2 and Q3 given in Sect. 7 intended as convergence criteria for the algorithm (Kułakowski 2015, Listing 1) are incorrect. Their values calculated under the assumptions of the example (Kułakowski 2015, Sect. 5) are also erroneous. Hence, the conditions and the values are corrected in this erratum.

Pairwise comparisons are widely recognized method supporting decision making process based on the subjective judgments. The key to this method is the notion of inconsistency that has a significant impact on the reliability of results. Inconsistency is expressed by means of inconsistency indices. Depending on their construction, such indices may pay attention to different aspects of the set of pairwise comparisons. The family of indices proposed in this article tries to combine the advantages coming from different indices, thereby increases the expressiveness of the family elements. The newly introduced notion of equivalence can help in comparing the indices and identifying their common properties.

Organization of a tender is not easy. Preparation of the relevant specification, taking into account the non-price criteria, implementation of the objective and fair assessment procedure, and last but not least, selecting a satisfactory offer are in practice a considerable challenge. In meeting this challenge appropriate multi-criteria assessment models can help. Models that can cope with different kinds of tangible and intangible criteria. The paper presents the hierarchical bid assessment (HBA) model of making decision in a tender procedure based on the pairwise comparisons method. It combines structural elements known from AHP with the Heuristic Rating Estimation approach. Two different schemes of rating tangible and intangible attributes are proposed. The notion of the success of the customer is defined and the practical method for its use is proposed. Theoretical considerations are illustrated in the relevant example.

Heuristic Rating Estimation (HRE) is a newly proposed method that supports decisions analysis based on the use of pairwise comparisons. It allows the ranking values of some alternatives (herein referred to as concepts) to be initially known, whilst ranks for other concepts have yet to be estimated. To calculate the missing ranks it is assumed that the priority of every single concept can be determined as the weighted arithmetic mean of the priorities of all the other concepts. It has been shown that the problem has an admissible solution if the inconsistency of the pairwise comparisons is not too high. The proposed approach adopts heuristics according to which a weighted geometric mean is used to determine the missing priorities. In this approach, despite increased complexity, a solution always exists and its existence does not depend on the inconsistency of the input matrix. Thus, the presented approach might be appropriate for a larger number of problems than previous methods. Moreover, it turns out that the geometric approach, as proposed in the article, can be optimal. The optimality condition is presented in the form of a corresponding theorem. A formal definition of the proposed geometric heuristics is accompanied by two numerical examples.

This paper presents an attempt at an analysis of parametric evaluation of research units with machine learning toolkit. The main goal was to investigate if the rules of evaluation can be expressed in a readable, transparent, and easy to interpret way. A further attempt was made at investigating consistency of the applied procedure and presentation of some observed anomalies.

Comparison, rating, and ranking of alternative solutions, in case of multicriteria evaluations, have been an eternal focus of operations research and optimization theory. Numerous approaches at practical solving the multicriteria ranking problem. The recent focus of interest in this domain was the event of parametric evaluation of research entities in Poland. The principal methodology was based on pairwise comparisons. For each single comparison, four criteria have been used. One of the controversial points of the assumed approach was that the weights of these criteria were arbitrary. The main focus of this study is to put forward a theoretically justified way of extracting weights from the opinions of domain experts. Theoretical bases for the whole procedure are based on a survey and its experimental results. Discussion and comparison of the two resulting sets of weights and the computed inconsistency indicator are discussed.

The Heuristic Ratio Estimation (HRE) approach proposes a new way of using the pairwise comparisons matrix. It allows the assump- tion that some alternatives (herein referred to as concepts) are known and fixed, hence the weight vector needs to be estimated only for the other unknown values. The main purpose of this paper is to extend the previously proposed iterative HRE algorithm and present all the heuris- tics that create a generalized approach. Theoretical considerations are accompanied by a few numerical examples demonstrating how the se- lected heuristics can be used in practice (Free PDF to download).

Lecture at McMaster University, Hamilton, Canada (04 September 2013)

click here to download

The pairwise comparisons method is a powerful tool used for establishing the relative order between different concepts in situations in which it is difficult (or sometimes even impossible) to provide explicit rating. Appropriate ratings are determined by solving eigenvalue problem for the pairwise comparisons matrix. This study presents a new iterative heuristic rating estimation algorithm that tries to deal with the situation when exact esti mations for some concepts (stimulus) $C_K$ are a priori known and fixed, whilst the estimates for the others (unknown concepts $C_U$) need to be computed. The relationship between the local estimation error, understood as the average absolute error $E\left(c\right)$ over all direct estimates for the concept $c\in C_U$ and the pairwise comparisons matrix inconsistency index is shown. The problem of convergence of subsequent intermediate results is discussed and the convergence conditions are given.

Is ready to use (no liability). Read more at PC Package