Published on November 2, 2016
1. Fact or Fiction? What Software Analytics Can Do For Us Andy Zaidman Delft University of Technology SPLASH-I 2016, Amsterdam azaidman
2. I like to tell stories… Andy
3. EVERYBODY LIES
4. EVERYBODY LIES
5. EVERYBODY Andy Zaidman
6. ANDY LIES
7. EVERYBODY Andy Zaidman Moritz Beller Georgios Gousios Annibale Panichella Niels Spruit
8. Let’s consider software testing
9. ANDY’S LIE Grant proposal “It is well-known that testing consumes 50% of a developer’s time”
10. “it depends” “50% of your time”
11. What is the (likely) origin of the 50%?
12. The Mythical Man-Month? Brooks “observed” that testing and debugging took 50% of the time How? “Calendar research” When ± 40 years ago…
13. ANDY’S LIE FACTS Grant proposal “It is well-known that testing consumes 50% of a developer’s time ”
14. Chinese whispers / telephone game
15. “Truth begins in lies” -House
16. Lies begin in truth “Truth begins in lies” -House
17. Folklore/evidence test approximately 50% Lack of knowledge How much time is actually spent on testing?
18. Research Tool Software Analytics “Big Brother” in your IDE measures testing activities • 416 Softw. Engineers, 68 countries • 5 month observation period • Java / Eclipse • 13.7 man years of development observed Immediate feedback for developers
19. WatchDog Pre-Test Questionnaire Can you estimate how much time you spend on engineering test code versus production code? 50% 50%test code production code 30% 70%test code production code
20. WatchDog Pre-Test Questionnaire Can you estimate how much time you spend on engineering test code versus production code? 48% - 52%test code production code
21. After measuring ≥ 3 months 25% - 75%test code engineering Most participants overestimate their test engineering activities production code engineering
22. Not all Java projects do unit testing Likely, an underestimation…
23. No test engineering For projects with test code: 53% of developer did not execute, read, or modify a single test within five months.
24. Do sets of unit test executions take… < 5 seconds ≥ 5 seconds
25. Swift tests Execution of set of unit tests takes 0.54 seconds (median).
26. Ad hoc test selection Developers frequently select a specific set of tests to run in the IDE. In most cases, developers execute one single test.
27. Test execution in the IDE is rare In 85% of development sessions no tests were run…
28. Tests fail 65% of test executions in the IDE fail.
29. Test Driven Development TDD is not widely practiced. Programmers who claim to do it, neither follow it strictly nor for all their modifications.
30. • Be critical: are long-standing beliefs/insights/claims • Still valid? • Valid in your context? • Triangulate • Developer opinions deviate greatly from what we objectively measured • Mixed-method research • Longitudinal studies
31. • Estimating is difficult • Estimating something that you don’t like is more difficult • Beliefs are to be questioned all the time
32. Differential diagnosis - Tests are written, but not run - Tests are written, but run on Command Line - Tests are written, but run in Continuous Integration
33. Differential diagnosis - Tests are written, but not run - (Tests are written, but run on Command Line) - Tests are written, but run in Continuous Integration
34. • 1,359 Java and Ruby projects • 2,640,825 builds
35. Testing in Continuous Integration In our population of 1,359 Java and Ruby projects
36. Builds with at least one failing test From 1,108 projects which executed tests
37. Testing #1 reason for build to break From 1,108 projects which executed tests Test failure Other failure
38. Tests are decisive for overall build From 1,108 projects which executed tests 98.3% projects Test fails Build breaks
39. Integrating in multiple environments From 1,108 projects which executed tests 60%projects Different build result for at least one build in multi-environment setup
40. Differential diagnosis - Tests are written, but not run - (Tests are written, but run on Command Line) - Tests are written, but run in Continuous Integration - 60% failing tests in the IDE - <20% failing tests in CI - Command Line?
41. ANDY’S LIES 1. 50% of time goes to testing 2. This talk was on software analytics
42. Empirical Software Engineering Software Analytics
43. Software analytics = bait / intrinsic motivation
44. From an experienced developer… “Estimated time working on tests: 20%, actual time: 6%. Cool statistics! ” Testing analytics helps developers to get a better understanding about their own development behavior!
45. Software analytics Empowers software engineers, software teams, managers Make better tools
47. WatchDog 2.0 We are looking into how developers debug Product improvement: In contact with Eclipse + IntelliJ New features coming to Eclipse based on our analytics Educational improvement: We need to teach our students to debug more efficiently (with the IDE debugger)
48. What to expect?
49. Software Analytics Challenges for the SPLASH community • What works in languages? What doesn’t? • How should we design the next version of languages? • How should we design the next generation of tools? • How should we improve education?
50. Software engineering folklore Perception w.r.t. dev. testing Measured developer testing 48% - 52% 25% - 75% Software analytics help engineers to better understand their own behavior azaidman WatchDogTeam Challenges for SPLASH
51. Pointers to literature 1. Moritz Beller, Georgios Gousios, Andy Zaidman: How (Much) Do Developers Test? ICSE 2015: 559-562 2. Moritz Beller, Georgios Gousios, Annibale Panichella, Andy Zaidman: When, how, and why developers (do not) test in their IDEs. ESEC/SIGSOFT FSE 2015: 179-190 3. Moritz Beller, Georgios Gousios, Andy Zaidman: Oops, my tests broke the build: An analysis of Travis CI builds with GitHub. PeerJ PrePrints 4: e1984 4. Fred Brooks: The Mythical Man-Month, Addison-Wesley, 1975 5. Laurent Bossavit: The Leprechauns of Software Engineering, 2012- 2014, https://leanpub.com/leprechauns
What software analytics can do for us. Session Program. Tue 1 Nov ... Fact or fiction? What software analytics can do for us (developers and researchers) Andy Zaidman.
... BI Platforms and Analytics Software Asia/Pacific and a Listed Vendor for Other Vendors to ... Original Source – Fact or Fiction? ... Contact Us; Support; Blog ...
This independence lets us ruthlessly call out fact from fiction in ... can and can’t do for you in the year ahead. Fiction: ... Software; Call Analytics ...
What Software Analytics Can Do For Us; Fact or Fiction? What Software Analytics Can Do For Us Feb 17, 2017 ... Share Fact or Fiction? What Software Analytics Can Do ...
Analytics; Bluewolf CEO on ... Apple Augmented Reality Glasses, fact or fiction? Increase / Decrease text size . Joe Clark ... Open Source Hippie to Software Standard
Help us to help you. Share your ... Fact or Fiction? ... the business user must be able to interact with their analytics software and build the story through guided ...
... let’s look at some SEO statements and find out if they’re fact or fiction! SEO fact or fiction? ... conversion and web analytics ... good for us, because ...
Fact or Fiction Find for Yourself ... I wonder who gets such fancy ideas to entertain us the ... This is a prevailing tool offered by Big Data services and Analytics ...
LinkedIn is het grootste zakelijke netwerk ter wereld en stelt professionals als Hennie Huijgens in staat om ... Liked "Fact or Fiction? What Software Analytics Can ...