Downloads provided by UsageCounts
Presentation code for the paper: Bayesian Performance Analysis for Algorithm Ranking Comparison. This repository contains the presentation code and tools used in the published paper. More recent versions can be found in: BayesPermus Package: https://github.com/ml-opt/BayesPermus Presentation code: https://github.com/ml-opt/BayesPermusPresentation Abstract In the field of optimization and machine learning, the statistical assessment of results has played a key role in conducting algorithmic performance comparisons. Classically, null hypothesis statistical tests have been used, however, recently, alternatives based on Bayesian statistics have shown great potential in complex scenarios, especially, when dealing with the uncertainty in the comparison. In this work, we delve deep into the Bayesian statistical assessment of experimental results by proposing a framework for the analysis of several algorithms on several problems/instances. To this end, experimental results for each experiment are transformed to their corresponding rankings of algorithms assuming that these rankings have been generated by a probability distribution (defined on permutation spaces). From the set of rankings, we estimate the posterior distribution of the parameters of the probability models considered, and several inferences concerning the analysis of the results are analysed. Particularly, we study questions related to the probability of having one algorithm in the first position of the ranking or the probability that one algorithm is in the ranking before another. Not limited to that, the assumptions, strengths, and weaknesses of the models in each case are studied. Finally, we provide some guidelines for the authors to avoid the misuse of the presented analysis.
Bayesian inference, probabilistic models on permutation spaces, benchmarking, evolutionary algorithms.
Bayesian inference, probabilistic models on permutation spaces, benchmarking, evolutionary algorithms.
| selected citations These citations are derived from selected sources. This is an alternative to the "Influence" indicator, which also reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically). | 0 | |
| popularity This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network. | Average | |
| influence This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically). | Average | |
| impulse This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network. | Average |
| views | 12 | |
| downloads | 3 |

Views provided by UsageCounts
Downloads provided by UsageCounts