Dynamic program analysis for suggesting test improvements to developers

Abstract

Automated testing is at the core of modern software development. Yet developers struggle when it comes to the evaluation of the quality of their test cases and how to improve them. The main goal of this thesis is precisely that, to generate concrete suggestion that developers can follow to improve their test suite. We propose the use of extreme mutation, or extreme transformations as an alternative to discover testing issues. Extreme transformations are a form of mutation testing that remove the entire logic of a method instead of making a small syntactic change in the code. As it traditional counterpart it challenges the test suite with a transformed variant of the program to see if the test cases can detect the change. In this thesis we assess the relevance of the testing issues that extreme transformations can spot. We also propose a dynamic infection-propagation analysis to automatically derive concrete test improvement suggestions from undetected extreme transformations. Our results are validated through the interaction with actual developers. We also report the industrial adoption of parts of our results. developers to improve their tests by detecting more of these transformations. Our results are validated through the interaction with actual developers.