Fact or Fiction? What Software Analytics Can Do For Us

58 %
42 %
Information about Fact or Fiction? What Software Analytics Can Do For Us

Published on November 2, 2016

Author: azaidman

Source: slideshare.net

1. Fact or Fiction? What Software Analytics Can Do For Us Andy Zaidman Delft University of Technology SPLASH-I 2016, Amsterdam azaidman

2. I like to tell stories… Andy

3. EVERYBODY LIES

4. EVERYBODY LIES

5. EVERYBODY Andy Zaidman

6. ANDY LIES

7. EVERYBODY Andy Zaidman Moritz Beller Georgios Gousios Annibale Panichella Niels Spruit

8. Let’s consider software testing

9. ANDY’S LIE Grant proposal “It is well-known that testing consumes 50% of a developer’s time”

10. “it depends” “50% of your time”

11. What is the (likely) origin of the 50%?

12. The Mythical Man-Month? Brooks “observed” that testing and debugging took 50% of the time How? “Calendar research” When ± 40 years ago…

13. ANDY’S LIE FACTS Grant proposal “It is well-known that testing consumes 50% of a developer’s time [1]”

14. Chinese whispers / telephone game

15. “Truth begins in lies” -House

16. Lies begin in truth “Truth begins in lies” -House

17. Folklore/evidence test approximately 50% Lack of knowledge How much time is actually spent on testing?

18. Research Tool Software Analytics “Big Brother” in your IDE  measures testing activities • 416 Softw. Engineers, 68 countries • 5 month observation period • Java / Eclipse • 13.7 man years of development observed Immediate feedback for developers

19. WatchDog Pre-Test Questionnaire Can you estimate how much time you spend on engineering test code versus production code? 50% 50%test code production code 30% 70%test code production code

20. WatchDog Pre-Test Questionnaire Can you estimate how much time you spend on engineering test code versus production code? 48% - 52%test code production code

21. After measuring ≥ 3 months 25% - 75%test code engineering Most participants overestimate their test engineering activities production code engineering

22. Not all Java projects do unit testing Likely, an underestimation…

23. No test engineering For projects with test code: 53% of developer did not execute, read, or modify a single test within five months.

24. Do sets of unit test executions take… < 5 seconds ≥ 5 seconds

25. Swift tests Execution of set of unit tests takes 0.54 seconds (median).

26. Ad hoc test selection Developers frequently select a specific set of tests to run in the IDE. In most cases, developers execute one single test.

27. Test execution in the IDE is rare In 85% of development sessions no tests were run…

28. Tests fail 65% of test executions in the IDE fail.

29. Test Driven Development TDD is not widely practiced. Programmers who claim to do it, neither follow it strictly nor for all their modifications.

30. • Be critical: are long-standing beliefs/insights/claims • Still valid? • Valid in your context? • Triangulate • Developer opinions deviate greatly from what we objectively measured • Mixed-method research • Longitudinal studies

31. • Estimating is difficult • Estimating something that you don’t like is more difficult • Beliefs are to be questioned all the time

32. Differential diagnosis - Tests are written, but not run - Tests are written, but run on Command Line - Tests are written, but run in Continuous Integration

33. Differential diagnosis - Tests are written, but not run - (Tests are written, but run on Command Line) - Tests are written, but run in Continuous Integration

34. • 1,359 Java and Ruby projects • 2,640,825 builds

35. Testing in Continuous Integration In our population of 1,359 Java and Ruby projects

36. Builds with at least one failing test From 1,108 projects which executed tests

37. Testing #1 reason for build to break From 1,108 projects which executed tests Test failure Other failure

38. Tests are decisive for overall build From 1,108 projects which executed tests 98.3% projects Test fails  Build breaks

39. Integrating in multiple environments From 1,108 projects which executed tests 60%projects Different build result for at least one build in multi-environment setup

40. Differential diagnosis - Tests are written, but not run - (Tests are written, but run on Command Line) - Tests are written, but run in Continuous Integration - 60% failing tests in the IDE - <20% failing tests in CI - Command Line?

41. ANDY’S LIES 1. 50% of time goes to testing 2. This talk was on software analytics

42. Empirical Software Engineering Software Analytics

43. Software analytics = bait / intrinsic motivation

44. From an experienced developer… “Estimated time working on tests: 20%, actual time: 6%. Cool statistics! ” Testing analytics helps developers to get a better understanding about their own development behavior!

45. Software analytics Empowers software engineers, software teams, managers Make better tools

46. Debugging

47. WatchDog 2.0 We are looking into how developers debug Product improvement: In contact with Eclipse + IntelliJ New features coming to Eclipse based on our analytics Educational improvement: We need to teach our students to debug more efficiently (with the IDE debugger)

48. What to expect?

49. Software Analytics Challenges for the SPLASH community • What works in languages? What doesn’t? • How should we design the next version of languages? • How should we design the next generation of tools? • How should we improve education?

50. Software engineering folklore Perception w.r.t. dev. testing Measured developer testing 48% - 52% 25% - 75% Software analytics help engineers to better understand their own behavior azaidman WatchDogTeam Challenges for SPLASH

51. Pointers to literature 1. Moritz Beller, Georgios Gousios, Andy Zaidman: How (Much) Do Developers Test? ICSE 2015: 559-562 2. Moritz Beller, Georgios Gousios, Annibale Panichella, Andy Zaidman: When, how, and why developers (do not) test in their IDEs. ESEC/SIGSOFT FSE 2015: 179-190 3. Moritz Beller, Georgios Gousios, Andy Zaidman: Oops, my tests broke the build: An analysis of Travis CI builds with GitHub. PeerJ PrePrints 4: e1984 4. Fred Brooks: The Mythical Man-Month, Addison-Wesley, 1975 5. Laurent Bossavit: The Leprechauns of Software Engineering, 2012- 2014, https://leanpub.com/leprechauns

Add a comment

Related pages

Fact or fiction? What software analytics can do for us ...

What software analytics can do for us. Session Program. Tue 1 Nov ... Fact or fiction? What software analytics can do for us (developers and researchers) Andy Zaidman.
Read more

Fact or Fiction? Smart Data Visualization Tells the Tale

... BI Platforms and Analytics Software Asia/Pacific and a Listed Vendor for Other Vendors to ... Original Source – Fact or Fiction? ... Contact Us; Support; Blog ...
Read more

Separate marketing technology fact from fiction at MarTech ...

This independence lets us ruthlessly call out fact from fiction in ... can and can’t do for you in the year ahead. Fiction: ... Software; Call Analytics ...
Read more

Fact or Fiction? What Software Analytics Can Do For Us ...

What Software Analytics Can Do For Us; Fact or Fiction? What Software Analytics Can Do For Us Feb 17, 2017 ... Share Fact or Fiction? What Software Analytics Can Do ...
Read more

Apple Augmented Reality Glasses, fact or fiction ...

Analytics; Bluewolf CEO on ... Apple Augmented Reality Glasses, fact or fiction? Increase / Decrease text size . Joe Clark ... Open Source Hippie to Software Standard
Read more

Fact or Fiction? Smart Data Visualization Tells the Tale ...

Help us to help you. Share your ... Fact or Fiction? ... the business user must be able to interact with their analytics software and build the story through guided ...
Read more

SEO fact or fiction? • Yoast

... let’s look at some SEO statements and find out if they’re fact or fiction! SEO fact or fiction? ... conversion and web analytics ... good for us, because ...
Read more

Fact or Fiction Find - This is the Business Intelligence Land

Fact or Fiction Find for Yourself ... I wonder who gets such fancy ideas to entertain us the ... This is a prevailing tool offered by Big Data services and Analytics ...
Read more

Hennie Huijgens | LinkedIn

LinkedIn is het grootste zakelijke netwerk ter wereld en stelt professionals als Hennie Huijgens in staat om ... Liked "Fact or Fiction? What Software Analytics Can ...
Read more