Published on January 7, 2009
Fast & Predictable – A Lightweight Release Framework Promotes Agility through Rhythm and Flow Amy Farrow; Steve Greene salesforce.com firstname.lastname@example.org; email@example.com Our answer to this problem was a lightweight Abstract Release Framework and the introduction of a Release Manager for the overall product release. Embarking on salesforce.com's large scale transformation to our Adaptive Development The result is a highly effective Technology Methodology presented many unique challenges due organization, operating on a consistent iteration and to the scale of the rollout as well as the need to align a release rhythm, enabling flow of information within large number of teams (30+) to a common release the organization, reducing cognitive dissonance, date, 3-4 times per year. To specifically address the realizing organic alignment with the rhythm, and latter challenge, we created a lightweight release optimization of the whole of the product release. framework in order to optimize on-time, high quality delivery to our customers and partners. 2. Release Framework: Guiding Goals & The fundamentals of this framework and approach Principles emphasize supporting and enabling the development teams rather than controlling them, through a release A very clear set of goals and principles provided us framework--simple, lightweight, unambiguous, visible guidance us in defining the Release Framework. everywhere. By implementing this lightweight release framework and making it highly visible, we've 2.1. Consistent, On-Time Delivery, No promoted rhythm and flow within and across all scrum Exceptions teams. In doing so, we have been able to develop a stable rhythm for delivering on-time, high-quality One of the founding principles of our ADM releases in a complex environment, while not implementation was that we would deliver on-time, compromising our Agile principles. high quality releases consistently to deliver value to our customers and partners. To achieve this end 1. Introduction required discipline within the management team to reject any requests for exceptions in the first two Salesforce.com initiated a transition to the Adaptive releases we delivered with ADM. This change Development Methodology, based upon scrum, in dramatically simplified the process and focused October 2006. Shortly after the initial rollout to 30+ Product Owners on the task of prioritizing effectively product teams it was clear that we had unleashed the to deliver as much value as possible in the timeframe power of our individual teams through their new found available to them. In addition to this, in order to deliver self-organization and focus on short sprints. However, frequently and to converge to a common release date, since we aggregate the work of many sprints into a the scope of the release and the scope of work for each single release it was also becoming clear that our team working in the release had to be negotiable. scrum teams were operating independently without respect for the context of the larger release. Herding In the process of doing this, it became readily 30+ teams working in the same codeline to a apparent that we needed to adopt a concept of “Scrum predictable time-boxed release date is very challenging Team Equality”, meaning that in the scope of the in any traditional development model let alone an agile release and schedule, the work that one team was doing development model where we value self-organization was no more important than another team’s. In the over centralized control. same respect, one feature, no matter how important,
would not cause us to delay a release of 30+ teams’ environment we were very sensitive to imposing heavy work that was ready to go. milestones on our teams. However, we still believed that there were key milestones that would align our teams to our on-time, high quality release objectives 2.2. Scale for Today, Tomorrow, 10x while enhancing the agility of our teams. Those milestones are as follows: Salesforce.com is a consistently and rapidly growing organization so we needed to create a framework that 1. Release Kickoff – Alignment from the Start: At the would enable us to scale our ability to successfully beginning of each release, which coincided with the manage a release that today includes 30+ scrum teams event of opening of the codeline, the Release but tomorrow will easily surpass 60 scrum teams and Operations team organizes an R&D wide release beyond. We want to have a model that could give 10X kickoff meeting. The purpose of the meeting is to scalability. At the time we embarked on the ADM provide visibility to the release plan for each team, rollout, there was not a lot of experience in the industry highlight dependencies, generate excitement about related to enterprise agility at scale. There was little what’s coming next and align the teams on the next professional advice available on how to scale an agile release rhythm. organization like Salesforce.com with so many teams working in the same codeline. At this meeting our most senior executives participate by talking about key business objectives of the release 2.3. Optimize the Whole of the Release including exciting new innovations and how those innovations will drive customer satisfaction. Their We needed to ensure that the process or framework we commitment to the success of the release is clear. put into place would promote visibility and optimization of the whole release, or of the suite of In addition, each of our Chief Product Owners present product functionality that we deliver to our customers the release plans for their scrum teams (features, and our partners. architecture and infrastructure work, refactoring, etc). In a short amount of time they are challenged to With such a large organization working on the same convey their plans concisely and effectively. The level codeline, it was imperative that our global priorities of information team members derive from the kickoff were consistent and clear to all scrum teams in order to enables them to be effective in planning for their own maintain quality of the release throughout the teams, dependencies they had on others or others had development cycle. The priorities were as follows: on them, and to foresee any possible impacts to plan their time more effectively. 1. Maintain existing functionality and quality of the overall codeline on a daily basis. All automation We value the importance of the organization getting a regression tests must pass at a 99% passing look into what is planned to enable them to effectively threshold on every checkin. communicate and work with each other, but we also 2. Building new functionality is secondary to valued the time that we had to contribute to this, so we codeline stability. time-boxed the kickoff to 1.5 hours. While each scrum team had to invest time to After the Release Kickoff, all release plans are maintain the quality of the codeline on a daily basis, available for all teams working on the release and are they also experienced benefits from it as well. publicly shared with the entire Technology organization. 9. Release Framework – A Blueprint 2. Feature Freeze The Release Framework we created became a blueprint for all major releases which enabled Product Feature Freeze marked the end of the product Owners to prioritize adequately and enabled teams to development sprints and the start of what we call the define realistic release plans. Release Sprint. All new product features are expected to meet our “definition of done” at the end of each 9.1. Light on Milestones sprint but also be potentially releasable at this milestone. This milestone also coincides with the Part of the release framework included the creation of a beginning of the Release Sprint. Between Kickoff and few key milestones. In our self-organizing, agile Feature Freeze, teams are clear on the goals they need
to meet but are able to self-organize to achieve their that release even though not necessarily involved in the goals in that timeframe. release in its earlier stages. This “handoff” of ownership to release managers late in the game 3. Release Freeze exposes weaknesses in the waterfall approach promoting an environment of blame and frustration This milestone marks the end of the Release Sprint and when problems occur and releases are delayed. means the overall release has moved from a potentially releasable state to a releasable state. To differentiate When we introduced the Release Manager role at the activities from the Product Development Sprints salesforce.com, we believed that it was important that and Release Sprints we clearly documented the the role be engaged from the inception of the release activities that did not contribute value when completed through to production deployment partnering with the each month and actually made more sense to only rest of the organization to ensure success. The Release execute once per release. Manager provides a framework within which self- organizing teams could be independent and agile while at the same get a clear understanding of the context of 4. Release the release with a simple escalation path if something was preventing them from achieving their goals, as The final important milestone is the release! related to the release. To achieve this Release Managers transformed away from a traditional role of This was the extent of the milestones we used to create “driver of the release” to a servant leader role, clearing a release framework to promote self organization in the path of systemic release issues that slowed down or achieving the milestones to deliver a high quality, on- halted team success. time release. The rest was left to the teams. It was important to us when we created the release 9.2. Rhythmic Cycles framework that we introduced the concept of this role but needed to reset expectations around it, since we In order to support teams in establishing and were creating a new definition for this role, with the understanding their velocity and what they would be following key characteristics and responsibilities: able to achieve in a given release, it was important to create a consistent, reusable template for our releases • Engage throughout the lifecycle of the release, such that they exhibited similar characteristics and from inception through to deployment. mimicked the iterative monthly cycles in iterative 4 • Promote regular release visibility and keep the month cycles. health, quality and release schedule highly visible to the organization. The following diagram depicts this rhythmic cycle: • Resolve systemic release issues that that slowed down or halted team success. Release Release • Promote team/product equality with respect to the Sprint Sprint Sprint Sprint Sprint Sprint overall release schedule. Jan Feb March April May June Release & Sprint Rhythm We initially equated this to role to “Scrum Master for the Release” and this sums up what we were aiming to The rhythmic cycle provided a mechanism for organic achieve – a Servant Leader who would uphold the self-organization across, within, and amongst teams values and principles of ADM and our Release around how to collaborate better on their work. All Framework to enable the organization to deliver. teams are on a monthly cycle with sprints and all releases are on the same 4 month cycle. This consistent rhythm at a daily, sprint and release level is 10. Release Framework – In Action very powerful for our organization reducing cognitive dissonance and eliminating waste across teams. Defining the Release Framework was an important step which led to determination of some of the 9.3. Release Manager as Servant Release mechanics in action that would enable and promote Leader this framework. At many non-agile software development companies, release managers own the “end game” or delivery of
automation results and incoming defect rates 10.1. Release Schedule Visibility normalized by scrum teams and number of developers and quality engineers. The analysis of aggregated data A simple method that we use to expand visibility of at a release level (with comparisons to our previous our release schedule is posting large visually striking releases) provides a higher level of visibility into posters of the overall schedule throughout our release trends and quality hot spots in the overall workspace. The posters clearly communicate to and codeline. This analysis has been useful for scrum are a quick reference to the entire organization on a teams and the management team to remove systemic daily basis as they walk through our workspace. These obstacles that are impacting all teams relative to the posters help teams understand how all the release overall release as well as providing a guide for pieces fit together, and to give better insight into the coaching. The Release Manager takes the lead on overall mechanism for delivery in the SaaS model. analyzing and providing this information along with This simple technique has significantly increased recommended action if any. All of this is necessary in awareness of the key dates, milestones and rhythm of a large scale agile environment, however, done with a the release. light touch to ensure that we don’t violate our principles of self-organization for teams. 10.2. Release Health & Quality Visibility 10.4. Communication The Release Manager promotes daily visibility of the overall release quality and status through “Daily Communication is a key element of our Adaptive Drumbeats” that are made available to the entire R&D Development Methodology. At the beginning of our organization. Thresholds are defined across any given transition, Scrum Team and Scrum of Scrums day in the release cycle and thresholds are also defined communication was effective in understanding both at monthly intervals when the release is expected to be progress and issues or blockers facing the teams. potentially releasable. However, since we had three Scrum of Scrums we found that there were challenges with communication Here’s a snapshot: across Scrum of Scrums as it related to the overall release. As a result, the model we adopted to increase effectiveness and communication across all teams working on the release was not for the teams to come to the Release Manager but rather for the Release Manager to go to the teams, through the weekly Scrum of Scrums. This became a forum for bi-directional The difference here is that the biggest challenge with information sharing and issue resolution. on-time delivery in large projects or environments where 30+ teams are working on a release is that the 10.5. Push Decision Making to the Team - complexity is high. Promoting regular visibility into Scrum Teams Decide when “Release Ready” aggregate release status while scrum masters promote visibility of individual team status ensured that we kept As with any release, there comes a time when we need both the local (team) and global (release) status in our to decide whether or not the release is ready to be purview at all times. This framework approach also deployed to customers. In a traditional approach, this enabled team agility and ownership in understanding decision is often made behind closed-doors at the their dependencies and impacts, rather than another executive level. We believed that this decision should team resolving them on their behalf. be pushed down to the individual Scrum Teams to promote even more agility. It was important for each 10.3. Release-by-Release Comparison Visibility of the Scrum Teams to contribute to the release readiness decision because they held the greatest level Weekly reporting of release-by-release comparisons of detail about readiness for each features. As such, add another dimension of understanding of the current we expanded our Sprint “Definition of Done” to create state of the release – both how the release itself is a Release “Definition of Done”. At the Release Freeze progressing day by day as well as how this release on a milestone, we expected each scrum team to weekly basis compared to previous releases, communicate to the Release Manager that they had met normalized to make the comparisons meaningful and the Release “definition of done” criteria and that their insightful. We regularly analyze aggregated bug debt,
product area was ready to be released. We do continue Development Methodology, we experienced tension to maintain release signoff and decisions at the with a few of the Product Owners who continued to executive level; however, the framework we have expect the release date to be negotiable if their features created promotes a shared level of responsibility were not completed to their expectations. Guided by between the individual scrum teams and the executive our desire to deliver value to our customers and team. partners on a regular basis (through time-boxed releases and variable scope), we upheld the organizational will and resisted these demands. 11. Challenges Executive level support and commitment to both our Adaptive Development Methodology and to our There were some challenges with creating this Release Framework were the keys to overcoming this lightweight Release Framework that are worth challenge. Over time, this issue dissipated as teams mentioning. understood that release dates were not negotiable. We’ve seen significant behavior change and this issue 11.1. Release Manager / Scrum Master Role and is no longer part of the discussion Confusion. 12. Conclusion When we first introduced the Release Manager role, there was some role confusion between the Release A lightweight Release Framework can be very useful Manager and the Scrum Masters for individual teams. in scaling at the enterprise level. By providing this Since both roles were new to the organization at the framework we have increased the level of agility and same time, it took some time for us to evolve the roles self-organization throughout our organization. On the and the relationships between them. Once we did and vast continuum from rigid structure and control to self- clarified the Scrum Master responsibility to the Scrum organization, the key to success is to constantly strive Team and the Release Manager responsibility to the for openness and flexibility (that comes from self- overall Release, we were able to achieve an effective organization) and to implement a simple, lightweight operating model within the organization. release framework that supports and enhances those principles of self-organization. Finding the right 11.2. Scrum Team Equality balance is an art form and depends upon cultural issues within the organization and requires a high level of The concept of Scrum Team Equality was a hard one trust. Finding that complementary balance is to grasp initially given the sheer number of teams and challenging yet worthwhile as it can unleash your Product Owner pride in the importance of their organization’s potential. particular product area and feature set. In the first two agile on-time releases we delivered with our Adaptive
Presentación que realice en el Evento Nacional de Gobierno Abierto, realizado los ...
In this presentation we will describe our experience developing with a highly dyna...
Presentation to the LITA Forum 7th November 2014 Albuquerque, NM
Un recorrido por los cambios que nos generará el wearabletech en el futuro
Um paralelo entre as novidades & mercado em Wearable Computing e Tecnologias Assis...
Fast & Predictable – A Lightweight Release Framework Promotes Agility through Rhythm and Flow Amy Farrow; Steve Greene salesforce.com afarrow@salesforce ...
Fast & Predictable A Lightweight Release Framework Promotes Agility through Rhythm and Flow
Fast & Predictable ' A Lightweight Release Framework Promotes Agility through Rhythm and Flow Presented by Steve Greene, Amy Farrow. Stage: Committing to ...
Fast & Predictable -- A Lightweight Release Framework Promotes Agility through Rhythm and Flow ... we created a lightweight release framework in order ...
Agile 2008 Conference. ... Fast & Predictable A Lightweight Release Framework Promotes Agility through Rhythm and Flow Autorzy.
Fast & Predictable A Lightweight Release Framework Promotes Agility through Rhythm and Flow on ResearchGate, the professional network for scientists.
Fast & Predictable A Lightweight Release Framework Promotes Agility through Rhythm and Flow. Amy Farrow, Steve Greene. Conference: ...
Integrating Scrum with the Process Framework at Yahoo! ... Fast & Predictable A Lightweight Release Framework Promotes Agility through Rhythm and Flow. 224 ...
Title Venue Description; Fast & Predictable ' A Lightweight Release Framework Promotes Agility through Rhythm and Flow: Agile 2008. Committing to Quality