Benchmarking and performance evaluation

Abstract

Benchmarking and performance evaluation are major concerns for developers and researchers. However, performance evaluation and comparison is a challenging task due to the difficulty in reproducing the experimental results. This problem can be addressed in two ways: on the one hand by using platforms that mitigate non-deterministic behaviors, on the other hand by using advanced statistical methods. We highlight these challenges in the context of the performance evaluation of 4 languages running on 8 Java Virtual Machines.