advertisement

Mutation Testing with PIT (Booster 2014, 2014-MAR-13)

67 %
33 %
advertisement
Information about Mutation Testing with PIT (Booster 2014, 2014-MAR-13)
Technology

Published on March 11, 2014

Author: filipvanlaenen

Source: slideshare.net

advertisement

© Computas AS 11.03.14 Mutation Testing with PIT Filip van Laenen Booster 2014 2014-03-13

2 © Computas AS 11.03.14 Agenda • Basics of mutation testing • Relation to other testing techniques • Example • Mutation testing techniques • Mutation testing tools • Personal experiences and recommendations

3 © Computas AS 11.03.14 Basics of Mutation Testing

4 © Computas AS 11.03.14 Mutation Testing in a Nutshell Seeking The Summoner @ The Daily WTF http://thedailywtf.com/Articles/Seeking-The-Summoner.aspx

5 © Computas AS 11.03.14 Mutation Testing in a Nutshell (cont'd)

6 © Computas AS 11.03.14 Mutation Testing in a Nutshell (cont'd)

7 © Computas AS 11.03.14 Mutation Testing in a Nutshell (cont'd) • Unit tests guard the source code • But who guards the guardians? • Do the unit tests cover all source code? • Lines? • Branches? • Paths? • Do the unit tests test the right things? Mutation testing tests the tests!

8 © Computas AS 11.03.14 Mutation Testing in a Nutshell (cont'd)

9 © Computas AS 11.03.14 Mutation Testing in a Nutshell (cont'd)

10 © Computas AS 11.03.14 Mutation Testing in a Nutshell (cont'd)

11 © Computas AS 11.03.14 Mutation Testing in a Nutshell (cont'd)

12 © Computas AS 11.03.14 Mutation Testing in a Nutshell (cont'd)

13 © Computas AS 11.03.14 Mutation Testing in a Nutshell (cont'd)

14 © Computas AS 11.03.14 Mutation Testing in a Nutshell (cont'd) int max(int a, int b) { return (a < b) ? b : a; } int max(int a, int b) { return (a <= b) ? b : a; }

15 © Computas AS 11.03.14 Mutation Testing in a Nutshell (cont'd)

“ 16 © Computas AS 11.03.14 Does Mutation Testing Work? In practice, if the software contains a fault, there will usually be a set of mutants that can only be killed by a test case that also detects that fault. Geist et. al., “Estimation and Enhancement of Real-time Software Reliability through Mutation Analysis,” 1992

“ 17 © Computas AS 11.03.14 Does Mutation Testing Work? (cont'd) Complex faults are coupled to simple faults in such a way that a test data set that detects all simple faults in a program will detect most complex faults. K. Wah, “Fault Coupling in Finite Bijective Functions,” 1995

18 © Computas AS 11.03.14 Does Mutation Testing Work? (cont'd) • “Generated mutants are similar to real faults.” • Andrews, Briand, Labiche, ICSE 2005 • “Mutation testing is more powerful than statement or branch coverage.” • Walsh, Ph.D. Thesis, State University of New York at Binghampton, 1985 • “Mutation testing is superior to data flow coverage criteria.” • Frankl, Weiss, Hu, Journal of Systems and Software, 1997

19 © Computas AS 11.03.14 Relation to Other Testing Techniques

20 © Computas AS 11.03.14 Relation to Other Testing Techniques • Unit tests • Test-Driven Development (TDD) • Test coverage • Static code analysis • Fuzz testing

21 © Computas AS 11.03.14 Relation to Other Testing Techniques

22 © Computas AS 11.03.14 Is Mutation Testing New? • R. Lipton, “Fault Diagnosis of Computer Programs,” 1971 • R. Lipton et. al., “Hints on Test Data Selection: Help for the Practicing Programmer,” 1978 • Historical obstacles: • No unit testing • No TDD • Inefficient implementation • Hence time-consuming • No integration with IDEs

23 © Computas AS 11.03.14 Practical Example of Mutation Testing

24 © Computas AS 11.03.14 Code Along… https://github.com/computas-fvl/booster2014 git clone https://github.com/computas-fvl/booster2014.git mvn clean site:site org.pitest:pitest-maven:mutationCoverage

25 © Computas AS 11.03.14 Mutation Testing Techniques

26 © Computas AS 11.03.14 Mutation Testing Techniques • Three aspects: • Mutation injection • Mutation types • Unit test selection per mutant • Key properties: • Efficiency • Performance

27 © Computas AS 11.03.14 Mutation Injection • Source code mutation • Binary code mutation • Caveats: • De-mutation • Compilation errors • Invalid binary code

28 © Computas AS 11.03.14 Mutation Types • Some mutations never change behaviour • Constants reused by unit tests • Log messages • Some mutations can change behaviour • Switching between < and ≠ • Switching between < and ≤ • Some mutations always change behaviour • Switching between < and ≥

29 © Computas AS 11.03.14 Mutation Types (cont'd) int max(int a, int b) { return (a < b) ? b : a; } int max(int a, int b) { return (a <= b) ? b : a; } int max(int a, int b) { return (a >= b) ? b : a; }

30 © Computas AS 11.03.14 Mutation Types (cont'd) for (int i = 0; i < 10; i++) … for (int i = 0; i != 10; i++) … for (int i = 0; i >= 10; i++) …

31 © Computas AS 11.03.14 Mutation Types Guaranteed to Change Behaviour • Negation of the comparison • Switching between = and ≠ • Switching between < and ≥ • Switching between > and ≤ • Negation of boolean conditions • Adding a ¬, ! or ~ • Shortcutting boolean conditions • Replacement with True or False *

32 © Computas AS 11.03.14 Unit Test Selection • Goal: find the unit test that “kills” the mutant • Selection aids: • Hints • Name/package matching • Code coverage tools • Automatic learning • Other heuristics

33 © Computas AS 11.03.14 Unit Test Selection (cont'd) • System: • 50 classes • 20 unit tests per class • 1 ms per unit test • Unit testing time: 50 × 20 × 1ms = 1s • 10 mutants per class: • Brute-force: 10 × 50 × 1s = 6m 20s • Educated: 10 × 50 × 20 × 1ms = 10s

34 © Computas AS 11.03.14 Unit Test Selection (cont'd) • System: • 500 classes • 20 unit tests per class • 1 ms per unit test • Unit testing time: 500 × 20 × 1ms = 10s • 10 mutants per class: • Brute-force: 10 × 500 × 10s = 13h 53m 20s • Educated: 10 × 500 × 20 × 1ms = 1m 40s

35 © Computas AS 11.03.14 Complexity • f: Number of function points • φ: Number of function points per class, ≥ 1 • τ: Number of unit tests per function point, ≥ 1 • μ: Number of mutants per function point, ≥ 1 • Brute-force: (f × τ) × (f × μ) = τ × μ × f² • Educated force: τ × μ × φ × f • Ideal: τ × μ × f

36 © Computas AS 11.03.14 Complexity (cont'd) • c: Number of classes • φ: Number of function points per class, ≥ 1 • τ: Number of unit tests per function point, ≥ 1 • μ: Number of mutants per function point, ≥ 1 • Brute-force: (f × τ) × (f × μ) = τ × μ × φ² × c² • Educated force: τ × μ × φ² × c • Ideal: τ × μ × φ × c

37 © Computas AS 11.03.14 Loops for (int i = 0; i < 10; i++) … for (int i = 0; i < 10; i--) …

38 © Computas AS 11.03.14 Infinite Loops • Terminate mutants that take too long to run • What's “too long”? • Ruins the performance • Can be hard to predict

39 © Computas AS 11.03.14 Other Problems • Recursion: • Stack overflows • Out of memory exceptions • Syntax errors • Segmentation faults

40 © Computas AS 11.03.14 Mutation Testing Tools

41 © Computas AS 11.03.14 Mutation Testing Tools • Java: • PIT • Jester (Nester, Pester) • Jumble • Ruby: • Mutant • Heckle • C#: • NinjaTurtles

42 © Computas AS 11.03.14 PIT • Java • Junit & TestNG • Maven or command-line • Operates on byte code • Large set of mutators • Also possibly equivalent mutators available • Highly configurable • Sensible defaults • http://pitest.org/

43 © Computas AS 11.03.14 PIT Mutators • Conditionals Boundary • Negate Conditionals • Remove Conditionals* • Math • Increments • Invert Negatives • Inline Constant* • Return Values • Void Method Calls • Non Void Method Calls* • Constructor Calls*

44 © Computas AS 11.03.14 PIT Sample Report

45 © Computas AS 11.03.14 Jester • Java • JUnit • Usually run from the command-line • Grester for Maven2 • Operates on source code • Runs all unit tests on all classes • Reporting and documentation could be better • http://jester.sourceforge.net/ • http://sourceforge.net/projects/grester/

46 © Computas AS 11.03.14 Jester Sample Report Overview

47 © Computas AS 11.03.14 Jester Sample Detailed Report

48 © Computas AS 11.03.14 Pester and Nester • Pester • Jester for Python • PyUnit • Nester • Port of Jester for C# • NUnit • Integrated with Visual Studio • But outdated… • http://nester.sourceforge.net/

49 © Computas AS 11.03.14 Nester Sample Report

50 © Computas AS 11.03.14 Jumble • Java • JUnit • Run from the command-line • Operates on byte code • Runs unit tests on a class • Reporting and documentation could be better • Claims to be faster than Jester • http://jumble.sourceforge.net/index.html

51 © Computas AS 11.03.14 Jumble Sample Report Mutating Foo Tests: FooTest Mutation points = 12, unit test time limit 2.02s .. M FAIL: Foo:31: negated conditional M FAIL: Foo:33: negated conditional M FAIL: Foo:34: - -> + M FAIL: Foo:35: negated conditional ...... Score: 67%

52 © Computas AS 11.03.14 Mutant • Ruby 1.9 and 2.0 • rspec2 • Usually run from the command-line • Good to-the-point reporting • Good performance • OK documentation • Active project • https://github.com/mbj/mutant

53 © Computas AS 11.03.14 Heckle • Ruby 1.8 • Doesn't work on Ruby 1.9 • Test::Unit and rSpec • Usually run from the command-line • Runs a set of unit tests on a class or a method • Good to-the-point reporting • Good performance • Virtually no documentation • http://rubyforge.org/projects/seattlerb/ • http://docs.seattlerb.org/heckle/

54 © Computas AS 11.03.14 Heckle Mutations • Booleans • Numbers • Strings • Symbols • Ranges • Regexes • Branches (if, while, unless, until)

55 © Computas AS 11.03.14 NinjaTurtles • .Net: • C# • Visual Basic/VB.NET • Any other .NET language code • Doesn't seem much alive • http://www.mutation-testing.net/

56 © Computas AS 11.03.14 NinjaTurtles Mutations • Sequence point deletion • Arithmetic operator substitution (*, /, +, -, %) • Bitwise operator substitution (&, |, ^) • Branch substituion (condition, always and never branch) • Conditional boundary substition (< and <=, > and >=) • Substitution of reads from variables, parameters and fields of the same type • Substitution of writes to variables of the same type

57 © Computas AS 11.03.14 Personal Experiences and Recommendations

58 © Computas AS 11.03.14 Experiences and Recommendations • Use mutation testing from day 1 • Start on a small code base • Keep number of unit tests per class low • Have small classes • Select a good tool • Configurable • Flexible • One that can output the mutant

59 © Computas AS 11.03.14 Experiences and Recommendations (cont'd) • Believe the tool • Or try to proof that the tool is wrong • Fix the problem • Don't turn mutation testing off • Embrace the “more than 100%” test coverage • Path coverage • Less code • More unit tests • More intelligent unit tests

60 © Computas AS 11.03.14 Improvements

61 © Computas AS 11.03.14 Improvements • Integration with more unit testing frameworks • Better unit test–source code mapping • Better heuristics • Parallellisation • Better reporting • IDE integration • Building tool integration

62 © Computas AS 11.03.14 Questions? Computas AS Lysaker Torg 45, pb 482 N-1327 Lysaker NORWAY Tel +47-67 83 10 00 Fax +47-67 83 10 01 Org.nr: NO 986 352 325 MVA www.computas.com Contact: fvl@computas.com @filipvanlaenen

Add a comment

Related presentations

Presentación que realice en el Evento Nacional de Gobierno Abierto, realizado los ...

In this presentation we will describe our experience developing with a highly dyna...

Presentation to the LITA Forum 7th November 2014 Albuquerque, NM

Un recorrido por los cambios que nos generará el wearabletech en el futuro

Um paralelo entre as novidades & mercado em Wearable Computing e Tecnologias Assis...

Microsoft finally joins the smartwatch and fitness tracker game by introducing the...

Related pages

PIT Mutation Testing

PIT is a state of the art mutation testing system, ... as we got our PIT mutation coverage up into the 90s I stopped paying much ... 24 Dec 2014 PIT 1.1.3 ...
Read more

Booster 2014, 12th-14th March 2014 | Lanyrd

Booster 2014 A SOFTWARE CONFERENCE FOR THE WHOLE TEAM. 12 ... The Booster conference is an IT conference for the whole development team, ...
Read more

SonarCommunity/sonar-pitest · GitHub

Description / Features. PIT is a mutation testing tool for java. You can check out the official pitest web site for more details on mutation testing and PIT.
Read more

Give your tests mutant powers with PIT [Part 1]

Mutation Testing basics: ... Labels: Java, Mutation Testing, PIT, Software Engineer, TDD, Testing ... 2014 (24) ...
Read more

Blog Archive - TheLadders Engineering Stories

Blog Archive 2015 Visual State Testing With Mock Data Aug 03 2015 posted in Architecture, ... 2014 Pipes, Tubes, ... Mutation testing with PIT: ...
Read more

First monkeys with customized mutations born : Nature News ...

30 January 2014. Article tools ... Twin cynomolgus monkeys born in China are the first with mutations in specific ... in order to boost mutation efficiency.
Read more

@Alex Victoor Architecte Technique à la Société Générale

Votre build avec PIT Build Mutation Test Résultat . ... Build Mutation Test . #pitest @Alex_Victoor PIT le mutation testing facile ... •Booster l ...
Read more