Published on February 15, 2014
© All rights reserved Want to use slides from this presentation? Ask for permission: email@example.com Making the Agile Leap to Continuous Deployment Ethan Ram / 15 Feb 2014 V1.1
Ethan Ram - Bio • Chief architect of Videobet • Leading Videobet’s R&D and OPs teams Agility efforts • As VP R&D of GameGround transformed the R&D to run in Continuous Deployment environment • Hacker in soul and a Windows Internals hobbyist 2/15/2014 © All rights reserved Find me here: • firstname.lastname@example.org • LinkedIn • The Null Terminator blog 2
Videobet • A Playtech division • Innovative Gaming Platform Provider – Established 9 years ago – 150 employees in Tallinn and Kiev – Operate in 9 countries – 60% of UK market • R&D – Front-end teams (C++) – Server teams (Java) – Game development teams 2/15/2014 © All rights reserved 3
Agenda #1 inciple Pr anifesto Agile M y to satisf ty i s a nd st priori ur highe r through early le O e ab e custom elivery of valu th us d ontinuo c e. softwar 2/15/2014 • • • • • • • © All rights reserved Why Continuous Deployment Setting the goals Deployment technics Development technics Making the leap Some unexpected results Q&A 4
Goal: Continuous Deployment Wix “From Jan ‘13 to Jun ’13 we did 1500 At SlideShare, instead of big launches, we Deployments, 470 A/B Tests, We have been doing continuous deployment release code all the time... it reduces risk. 200 Feature Toggles” at Plex since our inception in 1995. We Since we are putting out small changes at a Yoav Avrahami formalized the process with our own “deploy”time, if it breaks we know what it is and can button in 2003, and hit the 25k deployment roll back. mark less than two years thereafter. We Rashmi currently average between 50-100 deployments per day to our massive online Flickr "10+ Deploys Per Day" ERP system. WordPress.com - we’ve averaged about 16 Conover Steve of the exciting things… about Digg is Jerry Foster One product releases a day, every day for theOne of the exciting things… about Digg is last continuous deployment --when developers fix continuous deployment when developers fix four and a half years… Everyone in our athat or add new feature, there's no need bug company has access to a deploy button abug or add aanew feature, there's no need to wait for scheduled release… the to releases the latest checked in code to about wait for aascheduled release… the turnaround time for aachange drops turnaround time for change drops 400 production servers in our web tier in less dramatically. dramatically. than 30 seconds (across 3 data centers). Andrew Bayer Andrew Bayer Toni Schneider 2/15/2014 © All rights reserved 5
Major Release 2/15/2014 © All rights reserved 6
No. I Can’t Do That • “We’re starting this campaign next week and I need the product to do X” • “We need to stop showing this feature to non-US customers” • “I think this idea would work. When can we start A/B testing live?” • “I need to know if users are clicking on this link and if they see this error message” • “Forget about the other features. We have to go live by the end of the month!” 2/15/2014 © All rights reserved 7
2/15/2014 Continuous Deployment: Yes We Can! © All rights reserved 8
How to Streamline Deployment? • Speed-up Dev-ops • Lower risk of deploying something broken • Streamline feature implementation 2/15/2014 © All rights reserved ??? 9
Monitoring & Rollback • Must haves – real-time – Monitor that features are working, not just servers – Monitor the business results – Visualize and put it on the big screen – One-click rollback • Rollback: the ultimate risk mitigation – Prepare to fail! • Will the rollback work? – Rollback drills 2/15/2014 © All rights reserved 10
Deployment & Servers Setup Go from – 8 hrs downtime on upgrade • Physical machines • IP based network topology • Layered servers: web-apps, backend, cache, db • Feature servers • Each team has a different deployment setup and scripts • Complex database schema upgrades To – 8 mins deploy, no downtime • Virtual machines • DNS based network topology • All servers on each VM – DB too!!! • Same VM setup on all environments – With different config – With different data-set • One deployment script run on dev machine >> production – Target tag and environment name as input I T’S EASY! 2/15/2014 • Versioned mini schema migrations © All rights reserved 11
DB Versioning With Flyway DB Flyway DB makes you Agile! • Mini schema migrations kept with source code – Applying migrations added between v1 to v2 is easy – Roll-back of migrations supported built-in • Each developer has her own DB • Merging team-branch migrations is even simpler than merging code • Integrative with Maven/Ant/scripts and Java APIs – Server can upgrade/lower DB schema by itself • Data for unit-testing can be added and removed on the fly 2/15/2014 © All rights reserved 12
Staging on Production • Get real users to use new version ASAP – Call it Beta, Early Availability, Version Trial, Testbed • Deploy new version to part of the servers and direct it traffic of part of the user-base – – – – – Deploy to a subset of servers Run manual tests & automated tests Direct some traffic to new version Monitor! Measure! Deploy to the rest after some time • Not only for consumer-based internet products! 2/15/2014 © All rights reserved 13
Quality Assurance • Integration Test Automation – Focus first on covering main data flows – More main features, less details • Part of the Continuous Integration – Build > Unit-test > Integration Tests – All tests must pass to allow build promotion … not so easy… • Run the same tests on all environments – Developers’ & production environments too • Don’t give-up on manual testing • ‘Test frenzy’ as version signs-off – One hour session, structured, lead by QA – ALL employees join. The secretary too. 2/15/2014 © All rights reserved 14
R&D Kanban Board: CD Workflow BACKLOG Backlog SELECTED Selected • Bugs • Improvements. • Daily prioritization 2/15/2014 IN In PROGRESS Progress •Feature owner • Coding • Unit testing assigned •Task • Integration testing Breakdown •User stories •Design/gfx •Test plan RESOLVED Resolved VERIFIED Verified • Need verification • Drop build to QA • Automated integration testing • Manual testing TESTBED Testbed • Verified • Ready to be installed on TestBed • Optional step: install on TestBed • Regression Testing • Enforce WIP limits • One owner for each column • Feature owner responsible for feature end-to-end… • QA plan and test completion (no need for bugs!) • Unit testing and integration testing • Use DBA/HTML/FLASH and QA resources as needed • Test it on testbed/production © All rights reserved DONE Done • Running on production • Done / Not relevant 16
Developing on Trunk 2/15/2014 © All rights reserved 17
Use feature toggle flag 2/15/2014 © All rights reserved 18
Developing on the Trunk • Use feature toggle flags • Each dev has a full system - update, compile, run it all, test-it • Write backwards compatible code – Assume some servers/services do not have the new feature/code yet – DB fields may be missing – Mark older APIs with [deprecated/obsolete] • Dynamic Configuration – Central config, dynamically loaded – Plan for no app/system restarts Thor n Fr ee?!? • Schema changes are added to code as DB versioning scripts • Larger features that require a branch are rare! • Education: Do not break the trunk! 2/15/2014 © All rights reserved 19
Product: From Revolution to Evolution V2.2 V2.3 / R403 R417 2/15/2014 © All rights reserved 20
Leaping into CD • Initiative must come from high mgmt. – It’s a large investment… and risky money t ime • It’s not all or nothing >>> – First milestone: weekly release – Set a deadline and commit: “we start releasing once a week right after the next major release in September” • It’s a project to deliver – Multi-function task-force: devs, QA, deployment – Full time! – Plan a lot – detailed task list etc. • Training to all teams: process, tools • Decide on a CD weekly schedule – Tuesday afternoon is best time for a release • Make results visible early! – And celebrate success! 2/15/2014 © All rights reserved 21
The first CD release felt a bit like this… ..Just Go For It! 2/15/2014 © All rights reserved 22
Some Unexpected Results • … the sky did not fall on us • Excellent process transparency – – – – R&D stopped fighting with Product Mgmt. Team leaders stopped blaming each other Devs thought it’s much more fun Everyone got more involved • We managed to attract much better engineers • R&D managers thought its awesome! 2/15/2014 © All rights reserved 23
Want to use slides from this presentation? Ask for permission: email@example.com Q&A Thanks for contributing! Yoav Avrahami @ Wix Mike Mcgarr @ Excella Thanks! 2/15/2014 © All rights reserved 24
Presentación que realice en el Evento Nacional de Gobierno Abierto, realizado los ...
In this presentation we will describe our experience developing with a highly dyna...
Presentation to the LITA Forum 7th November 2014 Albuquerque, NM
Un recorrido por los cambios que nos generará el wearabletech en el futuro
Um paralelo entre as novidades & mercado em Wearable Computing e Tecnologias Assis...
Why Continuous Deployment; Setting the goals; Development using Kanban workflows; Some critical development technics; Making the leap; Some unexpected ...
Introduction-----Continuous Deployment is the natural evolution of continuous integration and delivery. It is the ultimate culmination of software ...
Continuous Delivery vs. Traditional Agile ... agile and CD is the approach to making ... on Continuous Delivery vs. Continuous Deployment ...
This article explores how agile development, continuous ... Continuous integration in agile ... in the limited availability of deployment ...
Continuous delivery Making ... Continuous delivery is about producing regular iterations of ... Automate your deployment process so you are forced ...
Continuous Delivery Vs. Continuous Deployment: ... » A more agile business. ... making continuous deployment impractical.
Continuous Deployment – Fact or Fiction? In this session I’ll describe my experience with making the biggest Agile “leap” in a consumer facing web ...
That’s where rapid deployment and continuous delivery become ... on how high or how far to leap? ... piece of technology before making an ...