Published on January 7, 2009
Agile 2008 Conference Dependency Management in a Large Agile Environment Eric Babinet Rajani Ramanathan salesforce.com salesforce.com firstname.lastname@example.org email@example.com Abstract 2. Team Structure Salesforce.com’s R&D organization has over 30 Scrum teams working simultaneously in a single Product development at salesforce.com is organized release code branch. This report highlights practices into three business units: Applications, Platform, and that salesforce.com has been using successfully to Core Infrastructure. There are 30 plus Scrum teams scale Scrum and to manage inter-team dependencies. distributed across these three business units. Each Scrum team typically has dedicated developers, quality assurance engineers and a Product Owner. Each team 1. Introduction also has a ScrumMaster, who is usually a program manager, development manager, or QA manager. In October 2006, salesforce.com's R&D Depending on the complexity of the features being organization embarked on a large scale transformation developed, the Scrum team may also have dedicated or from a waterfall software development approach to an part-time team members from other functional teams agile approach based on Scrum. It had been 10 months such as System Testing, Documentation, UI Design, since the last major release, the planned release date of Usability, Technology Operations, and Release the next release had moved five times, and many Engineering. people were frustrated by the inability to deliver Typically all Scrum teams are working on the same frequently and on schedule. Instead of waiting until the major release and in the same codeline/branch. Given release was completed, we reconfigured existing teams the integrated nature of our products, the features into Scrum teams and used the Scrum process to implemented by these various teams can be very deliver the in-progress release to our customers in tightly interwoven, both functionally and technically. February 2007. Since then we have delivered five From a technical implementation perspective many major releases (at 3-4 month intervals) of our teams may be using common shared code. From a Software-as-a-Service (SaaS) application suite and functional perspective, teams working on features Force.com platform using our new agile approach. targeting the same end user need to coordinate closely Every release has gone into production exactly on the to provide a consistent and coherent user experience. pre-planned release date. During the transition, Scrum offered significant 3. Why is Dependency Management guidance for individual teams, but not a lot with Difficult? respect to coordinating between teams. We organized teams to minimize dependencies, but the code hadn’t With so many teams working together on the same changed overnight and there were still significant release and on shared features and code, it is dependencies between teams. We implemented scrum- impossible to anticipate all the issues, surprises, of-scrums meetings early and they were helpful for changes, failures and successes that teams will discussing issues and status, but not sufficient. Over encounter during software development. That the last five releases we've tried out and refined a unpredictability is one reason that dependency number of additional practices to improve coordination management is difficult. This section highlights some between teams. This report highlights challenges we’ve other reasons why dependency management is encountered with respect to dependency management difficult. Any solution must address these underlying in a large agile environment and how we overcame tensions and challenges. them. 978-0-7695-3321-6/08 $25.00 © 2008 IEEE 401 DOI 10.1109/Agile.2008.58
The ability of teams to change priorities each sprint is a benefit of Scrum, however, it can make managing 3.1. System Complexity dependencies more difficult. Dependency identification and analysis cannot just happen once at the beginning The first step in managing dependencies is to of the release, but rather must be an ongoing process. identify them. Like most mature software, Over the course of a release dependencies come and go salesforce.com's systems are complex enough that one as teams refine what they can deliver, and any process person or team cannot possibly know about all for managing those dependencies needs to be equally dependencies. Increasingly we must rely on the dynamic. collective knowledge of many people to identify and understand dependencies. Pooling this collective 3.4. Short and Overlapping Release Cycles knowledge and connecting those who have the knowledge with those that need the knowledge is Short release cycles mean that dependencies and essential. Experts with broad knowledge of the system impact must be understood early as there is less time to are highly valued and play a key role in identifying the coordinate with other teams. Salesforce.com’s release dependencies and impact of proposed changes. cycles are designed with some overlap, meaning that However, as the system grows in complexity it is hard the planning process for the next release starts before to avoid more specialization and knowledge the prior release is completely finished. This is fragmentation. Given the volume of change it is also intentional as some teams are completely finished and no longer practical for these experts to review the need to be able to move on to the next release. detail of every feature being developed. We need a However, teams that are not finished may fall behind method to identify changes with the highest likelihood in their release planning and be less available to of impacting or depending on other teams, so we can discuss next release dependencies with other teams. focus our attention on those areas. Of course this is not This is problematic given the importance of collective easy, as deceptively small changes can have a big knowledge in understanding dependencies. impact. 4. Specific Practices 3.2. Conflicting Priorities Dependencies can lead to conflict between teams. If This section describes the specific practices we are one team needs another team to do something, the using to manage dependencies and address the above product owners for those teams must negotiate the challenges. Some common strategies incorporated into relative priority of that work. If they negotiate these practices include: effectively and reach a common understanding for • Create visibility: for release plans, when the work can be done, then managing that dependencies, designs, build status, etc. dependency should be relatively easy. If they don’t • Provide forums to stimulate collaboration and reach a clear agreement, or if the work is still a lower knowledge sharing between teams. priority for the other team, there will be greater • Promote self-organization and decentralized uncertainty and risk around that dependency. dependency management. Another type of conflict occurs when one team • Automation, automation, and more automation. makes a change that imposes work on another team. A common example is when one team does architectural 4.1. Release Kickoff refactoring that requires supporting changes by other teams. The impacted teams may get no short-term About one month prior to the current major release benefit from these changes, and may resent having to going live in production, we hold a release kickoff make them, especially if the need for them is meeting for the next major release. This is an all-hands unexpected. We attempt to identify these types of one hour meeting in which the Senior Vice President changes early in the release, but inevitably there are for each business unit gives a high level presentation of some that show up later in the release, either because the features that each team is planning to build in the they arise opportunistically or due to a change in next release. This meeting has a number of benefits priorities. with respect to dependency management. First and foremost, it motivates all teams to complete their initial 3.3. Dynamic Scope release plan by a specific date. If a team has leftover work from the current release they can fall behind in planning for the next release, but the Release Kickoff 402
helps teams stay on track with that planning. It is each Scrum team in a room with a large white board. difficult to identify dependencies or negotiate Usually the representative is the ScrumMaster or commitments with other teams if those teams don't yet Product Owner from the team, but it can be anyone. know what they're doing in the release. The Release The first part of the meeting involves all team Kickoff meeting acts as an important synchronization representatives going up to the whiteboard point for teams, and helps ensure more productive simultaneously and creating two diagrams representing discussions around inter-team dependencies. the dependencies between teams. One diagram The second major benefit results from the captures teams that need another team to do something visibility that the meeting creates around team release for them. The second diagram captures teams that are plans. Our goal is to generate a high level of awareness doing something that will impact other teams. For of what teams are doing in the release, as we believe these diagrams we use a simple notation: that this visibility will naturally cause people to think • A circle represents a team about and discuss dependencies. In order to encourage • An arrow between two circles represents a attendance and focus attention on the release plans, the dependency. On the first diagram the arrow meeting is given a high-profile and all R&D executives points from the team doing the work to the team attend. The high level of participation helps align that needs the work done. On the second expectations and create broad awareness of what is diagram the arrow points towards the team that planned for the release. is impacted. Of course, team release plans can and do change • A label on the arrow describes the dependency throughout the release. We recently started reviewing • If a dependency is committed (i.e. the other an updated version of the Release Kickoff presentation team has agreed to do the work) we put a box during the monthly sprint reviews. The updated around the dependency label. presentation highlights features that have been dropped or added to the release, and has proven to be very At first there is a commotion as everyone draws effective at maintaining release plan visibility their dependencies on the whiteboard and onlookers throughout the release. start asking questions and pointing out dependencies that are missing. After about 20 minutes the diagram 4.2. Dependency Identification Exercise stabilizes and comments subside. At that point we ask each team representative to briefly describe their Once teams have their initial release plans, they can dependencies. have a more meaningful discussion about In a very short amount of time we generate a dependencies. There is a lot of informal discussion comprehensive view of the dependencies in the release. between teams to negotiate dependencies, but we have Figure 1 shows the typical output of this meeting after found it very valuable to facilitate a more formal it has been copied into a more readable format. Hot- Dependency Identification exercise. This is a highly spots (i.e. teams with a lot of dependencies) are interactive and collaborative event that involves immediately visible. Teams that have not yet obtained representatives from all teams. We have found it most commitments for their dependencies are also visible. effective to do the exercise shortly before the Release These teams have a higher risk of not delivering on Kickoff as it improves the quality of the release plans their release plans until the dependencies are presented at the Release Kickoff. committed. The actual logistics of the Dependency Identification exercise are simple. We gather representatives from 403
Figure 1. Example team dependency diagram organization we are experimenting with some 4.3. Release Open Space variations to see what works best, including: We recently started holding an open space style • Generating topic ideas prior to the open space meeting during the week after the Release Kickoff. The • purpose of this open space is to create a forum for Encouraging senior management participation discussing questions or concerns regarding team • Changing the number and length of breakout release plans. We ask that at least one representative sessions from each Scrum team attend, but otherwise the meeting is optional. In standard open space style, 4.4. Functional Design Reviews attendees propose topics at the beginning of the meeting. Then we have 45 minutes for breakout The goal of the functional design review meetings is discussions and 30 minutes for reports from the to improve the overall quality and usability of our breakout groups. Popular topics include wanting to products by: know more about functionality that is new and which • leveraging the collective wisdom and creativity can benefit other teams, and wanting to know more of our product teams about changes that will impact other teams. Attendees • improving design coherence across products have reported the open space to be educational and • creating synergies through cross-team have found it interesting to be in discussions with knowledge sharing people with whom they do not normally associate. Since the open space meeting format is new to the 404
These are cross-functional and cross-scrum team least 20% of their time every release towards meetings that focus on the functional and user implementing changes recommended by the VAT. interaction design of features deemed to be higher risk, The VAT meets for two hours twice a week to higher complexity, or higher impact. These meetings review the technical implementation of products and are intended to break down team silos and surface features being built by the Scrum teams. The teams design inconsistencies and dependencies of which the building the most complex features in the release are Scrum team may be unaware. asked to present to the VAT. The group provides There are two different parts to the functional valuable feedback to the Scrum team on how their design review meetings. Early in the release cycle the technical design will impact or be impacted by other discussions focus on design concepts and aim at areas. The VAT focuses primarily on the technical achieving functional design coherence and synergy implementation, especially scalability and performance between existing and new features. These discussions considerations. Teams asked to make significant are usually attended by the product owners of each changes must present again in the same release cycle Scrum team and UI designers, and there is little and provide details on how they modified their design. representation from other functional teams such as development or QA. 4.6. Continuous Integration Starting near the middle of the release cycle the participants change to primarily include QA and some We have a web-based automated build, test and development team members, and the focus is on triage infrastructure that provides a continuously reviewing implementation details. These discussions integrated build system and allows us to track the provide insight and clarity around: health of any given codeline. This infrastructure is • Additional dependencies between teams critical for enabling all developers and QA engineers to • Additional integration testing that needs to be work in a shared clean codebase. added to the test plan The primary test suite is built using an extension of • The release risk of a particular feature, and the JUnit framework and provides a mechanism for perhaps the need to restrict availability of that implementing functional tests through the Force.com feature API. We also have a UI testing framework using • Deployment specific tasks Selenium for automating test cases that need to be executed using the UI. Some fundamental tenets guiding our automation 4.5. Virtual Architecture Team approach are: 1. Provide fast feedback to developers so they The Virtual Architecture Team (VAT) is virtual understand the effects of their changes. Each check- as it is made up of developers from every Scrum team. in triggers a build and a subset of tests to be run. Build Members work on the VAT in addition to being on a failure notifications are sent to the responsible Scrum team from the Application, Platform or Core engineers immediately. The basic test suite runs within Infrastructure business units. half an hour and the developer is notified of the results The VAT owns maintaining and extending our of their change. Extended test suites run periodically industry-leading software architecture. They do this by throughout the day. defining the architectural road map, by reviewing 2. Fix build and test failures quickly. To ensure architecturally significant changes to the code, and by this, we have a rotating “build master” role that defining coding standards to ensure consistent and changes each week. This role is typically filled by two maintainable code. people, a dev manager and a senior developer, who The VAT maintains a backlog of major architectural coordinate closely. The build master is responsible for projects or refactoring changes that need to be monitoring build results, tracking test history, tracking implemented. As the product keeps evolving, we need failures across different code lines, and assigning bugs to redesign features and remove old code that is no to fix failures. The test results are logged in a database longer optimal. Developers are sometimes reluctant to and metrics are collected on a central dashboard (% of tamper with programs that already work, even when tests passing, build times, number of failures, etc.) The they know it would be better if coded differently. build master also uses this data to send out status Product Owners share their reluctance as they would emails. rather see new features developed than have something 3. High automated test coverage. Scrum teams that already works be rewritten. To counteract these typically aim to have over 70-80% of code covered by tendencies, the Application, Platform, and Core automated tests. Our large collection of automated tests Infrastructure business units are asked to allocate at 405
is one of our biggest assets and a key enabler for 5. Conclusion delivering high quality on-time releases every 3-4 months. Salesforce.com has demonstrated that it is possible to scale Scrum and we feel very confident in our ability 4.7. Scrum-of-Scrums to continue scaling as the organization grows. Dependency management and cross-team coordination Each business unit has a weekly scrum-of-scrums are a significant challenge as the number of teams meeting and there is also a weekly scrum-of-scrum-of- increases. Having a robust build and automated test scrums. Occasionally we will form additional scrum- infrastructure is absolutely critical. Increasing visibility of-scrums if there is a cluster of teams working closely and alignment through practices such as the release together on a common goal. We have experimented kickoff, the dependency identification exercise, scrum- with a few different formats for the scrum-of-scrums. of-scrums, and lightweight status reports is very We started out using the standard 4 question format helpful. Ultimately it comes down to effective where teams report on what they've done, what they're communication, collaboration, and knowledge sharing going to do, what is blocking them, and if they are between teams. Practices such as the virtual doing anything that will impact another team. That architecture team, the functional design reviews, the worked fine initially, but became a bit tedious due to release open space, and the scrum-of-scrums are all the large number of teams. We have since moved to a designed to open communication and promote more open and self-organizing format, in which collaboration. attendees propose discussion topics by writing them on the whiteboard at the beginning of the meeting. This approach leads attendees to take responsibility for the content of the meeting and has resulted in more productive and collaborative discussions. The topic of cross team dependencies does come up during the scrum-of-scrums, especially early in the release cycle. The Dependency Identification Exercise is held during a scrum-of-scrums meeting and for the next two to three weeks after that we will typically discuss changes to dependencies during the scrum-of-scrums. 4.8. Status Reports Written status reports are often discarded when a team moves to Scrum as visibility into team status is provided by other means (e.g. sprint review, sprint burndown chart, daily standup). When we adopted Scrum we decided to keep a lightweight status report written weekly by each ScrumMaster. When we were using the 4 question format for the scrum-of-scrums there was definitely some redundancy between the team's scrum-of-scrums update and the team's status report. However, now that we use the open format scrum-of-scrums, the weekly status report is actually an important complement to the scrum-of-scrums. We don't discuss individual team status in the scrum-of- scrums unless specifically raised as a discussion topic, however, the status is always available in the weekly report. Dependencies, blockers, and risks are all highlighted in the report. While the report is a little extra work for ScrumMasters we feel that the visibility it provides is worth the cost. 406
Dependency Management in a Large Agile Environment Presented by Eric Babinet, Rajani Ramanathan. Stage: Leadership & Teams. Session type: Experience report
Salesforce.com’s R&D organization made a comprehensive large scale transition to agile software development in October 2006. Since that time we have ...
Similar Documents. Urban Environment and Environmental Health Environmental Consultants For Sustainable Environment Top Three Environmental Diseases
IEEE membership options for an individual and IEEE Xplore subscriptions for an organization offer the most affordable access to essential journal articles ...
Salesforce.com’s R&D organization has over 30 Scrum teams working simultaneously in a single release code branch. This report highlights practices that ...
Dependency Management in a Large Agile Environment Babinet, E.; Ramanathan, R. Salesforce.comâ s R&D organization has over 30 Scrum teams working ...
Publication » Dependency Management in a Large Agile Environment. ... [Show abstract] [Hide abstract] ABSTRACT: This qualitative study investigates ...
... Ramanathan, R.: Dependency management in a large agile environment ... Reduced uncertainty through human communication in complex environments. Cogn ...