DevOps on AWS: Deep Dive on Continuous Delivery and the AWS Developer Tools

50 %
50 %
Information about DevOps on AWS: Deep Dive on Continuous Delivery and the AWS Developer Tools

Published on July 12, 2016

Author: AmazonWebServices

Source: slideshare.net

1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. James Saull, Solutions Architect – Amazon Web Services Emerson Loureiro, Senior Software Engineer - Gilt Groupe July 2016 DevOps on AWS: Deep Dive on Continuous Delivery and the AWS Developer Tools

2. https://secure.flickr.com/photos/mgifford/4525333972 Why are we here today?

3. Software moves faster today Software creation and distribution is easier and faster than ever: • Startups can now take on giants with little to no funding ahead of time • Getting your software into the hands of millions is a download away • Your ability to move fast is paramount to your ability to fight off disruption

4. What tools do you need to move fast? Releasing software in this new software driven world requires a number of things: • Tools to manage the flow of your software development release process • Tools to properly test and inspect your code for defects and potential issues • Tools to deploy your applications

5. First, we need to understand a little bit about software release processes https://www.flickr.com/photos/jurvetson/5201796697/

6. Release processes levels Source Build Test Production Continuous integration Continuous delivery Continuous deployment

7. Continuous Delivery Benefits Improve developer productivity Find and address bugs quickly Deliver updates fasterAutomate the software release process

8. A look back at development at Amazon.. https://secure.flickr.com/photos/pixelthing/15806918992/

9. 2001 Development transformation at Amazon: 2001-2009 2009 monolithic application + teams microservices + 2 pizza teams

10. In 2009, we ran a study to find out where inefficiencies might still exist

11. We were just waiting. WaitWrite Code WaitBuild Code WaitDeploy to Test Deploy to Prod Weeks Mins Days Mins Days Mins Days Mins

12. Automated actions and transitions; from check- in to production Development benefits: • Faster • Safer • Simplification & standardization • Visualization of the process Pipelines

13. This has continued to work out really well: In 2014: • Thousands of service teams across Amazon • Building microservices • Practicing continuous delivery • Many environments (staging, beta, production) 50 million deploys

14. Continuous delivery service for fast and reliable application updates Model and visualize your software release process Builds, tests, and deploys your code every time there is a code change Integrates with 3rd party tools and AWS AWS CodePipeline

15. Source Source GitHub Build JenkinsOnEC2 Jenkins Deploy JavaApp Elastic Beanstalk Pipeline Stage Action Transition CodePipeline MyApplication

16. Source Source GitHub Build JenkinsOnEC2 Jenkins Deploy JavaApp Elastic Beanstalk NotifyDevelopers Lambda CodePipeline MyApplication Parallel actions

17. Source Source GitHub Build JenkinsOnEC2 Jenkins Deploy JavaApp Elastic Beanstalk NotifyDevelopers Lambda TestAPI Runscope CodePipeline MyApplication Sequential actions

18. 8. Retrieve build artifact EC2 instance CodePipeline Source Source GitHub Build JenkinsOnEC2 Jenkins Deploy JavaApp Elastic Beanstalk Source Artifact S3 Build Artifact S3 5. Get source artifact 1. Get Changes 6. Store build artifact 3. Poll for Job 4. Acknowledge Job 7. Put Success 9. Deploy build artifact Elastic Beanstalk Web container Java App MyApplication

19. We have a strong partner list, and it’s growing Source Build Test Deploy

20. AWS service integrations Source Invoke Logic Deploy AWS Elastic BeanstalkAmazon S3 AWS CodeDeploy AWS Lambda AWS CodeCommit

21. DEMO https://www.flickr.com/photos/seattlemunicipalarchives/12504672623/

22. Deploying your applications https://secure.flickr.com/photos/simononly/15386966677

23. Automates code deployments to any instance Handles the complexity of updating your applications Avoid downtime during application deployment Deploy to Amazon EC2 or on-premises servers, in any language and on any operating system Integrates with 3rd party tools and AWS AWS CodeDeploy

24. appspec.yml Example version: 0.0 os: linux files: - source: / destination: /var/www/html permissions: - object: /var/www/html pattern: “*.html” owner: root group: root mode: 755 hooks: ApplicationStop: - location: scripts/deregister_from_elb.sh BeforeInstall: - location: scripts/install_dependencies.sh ApplicationStart: - location: scripts/start_httpd.sh ValidateService: - location: scripts/test_site.sh - location: scripts/register_with_elb.sh

25. appspec.yml Example version: 0.0 os: linux files: - source: / destination: /var/www/html permissions: - object: /var/www/html pattern: “*.html” owner: root group: root mode: 755 hooks: ApplicationStop: - location: scripts/deregister_from_elb.sh BeforeInstall: - location: scripts/install_dependencies.sh ApplicationStart: - location: scripts/start_httpd.sh ValidateService: - location: scripts/test_site.sh - location: scripts/register_with_elb.sh • Remove/add instance to ELB • Install dependency packages • Start Apache • Confirm successful deploy • More! • Send application files to one directory and configuration files to another • Set specific permissions on specific directories & files

26. v2 v2 v2 v2 v2 v2 one at a time half at a time all at once v2 v2 v2 v1 v1 v1 v2 v1 v1 v1 v1 v1 Agent Agent Dev Deployment group OR Prod Deployment group Agent AgentAgent Agent Agent Agent Choose Deployment Speed and Group

27. https://www.flickr.com/photos/spacex/16510243060/ Launching to Production

28. Launching to Production After you’ve built and tested your code and hopefully gone through a few pre-production deploys, its time for the real thing! You’ll want think about: • Impact to customers • Impact to infrastructure • Impact to business How can we track these and communicate deploys?

29. Extend AWS CodePipeline Using Custom Actions Update tickets Provision resources Update dashboards Send notifications Security scan Mobile testing

30. Source Source GitHub Build JenkinsOnEC2 Jenkins Deploy Action Custom Action JavaApp Elastic Beanstalk MyApplication CodePipeline Deploy JavaApp Elastic Beanstalk Job Worker 3. Perform Job 1. Poll for Job 2. Acknowledge Job 4. Put Success Amazon EC2 With custom actions, the job worker drives the interaction between AWS CodePipeline and other applications or services

31. Source Source GitHub Build JenkinsOnEC2 Jenkins Deploy MyAction AWS Lambda JavaApp Elastic Beanstalk 2. Perform Job 1. Invoke Lambda function 3. PutJobSuccessResult MyApplication CodePipeline With AWS Lambda-based actions, AWS CodePipeline drives the integration with Lambda, which then connects with other applications or services AWS Lambda

32. Micro-services Deployment at Gilt with AWS Emerson Loureiro, Senior Software Engineer, Gilt

33. Gilt... ●Sales go live everyday at noon

34. Micro-services

35. Move fast

36. Phased rollout

37. Rollback just as fast

38. Deployment tooling is important for us... Can we do it with AWS?

39. 1st Generation (DiscoDeploy) ● SBT plugin awsAccountName: gilt-xxxxxxx instanceType: m3.xlarge servicePort: 9000 instanceCount: 2 jvm: - -Dnewrelic.config.file=/opt/gilt/service/conf/newrelic.yml - -Dnewrelic.config.agent_enabled=true - -Dlogger.resource=logback.production.xml - -Dconfig.resource=application.production.conf - -Xms12G - -Xmx12G - -XX:NewSize=6G - -XX:MaxNewSize=6G ...

40. 1st Generation (DiscoDeploy) Docker Registry ASG EC2 EC2 EC2 EC2 ELBsbt deploy-setup sbt release-docker sbt deploy-service sbt deploy-scale

41. ●Good to begin with 1st Generation (DiscoDeploy) Docker Registry ASG EC2 EC2 EC2 EC2 ELB

42. 2nd Generation (Ionblaster) ●Based on CloudFormation [ { "service": "SERVICE", "domain": "SERVICE.gilt.com", "instances": { "min": 3, "max": 3 }, "type": "t2.small" "run": { "image": "gilt/SERVICE", "cmd": ["-Dhttp.port=9000"], "args": ["-p", "9000:9000"] }, "health_path": "_internal_/healthcheck" ... } ]

43. ●Based on CloudFormation sbt docker:publish ionblaster new SERVICE 0.0.1 2nd Generation (Ionblaster) 0.0.1 DNS

44. 2nd Generation (Ionblaster) ●Based on CloudFormation sbt docker:publish ionblaster new SERVICE 0.0.2 ionblaster traffic SERVICE 0.0.2 10 0.0.1 DNS 0.0.2

45. 2nd Generation (Ionblaster) ●Based on CloudFormation ionblaster traffic SERVICE 0.0.2 100 0.0.1 DNS 0.0.2

46. 2nd Generation (Ionblaster) ●Based on CloudFormation ionblaster kill SERVICE 0.0.1 DNS 0.0.2

47. ●Traffic migration could be better ●No logical grouping among stacks 2nd Generation (Ionblaster) DNS 0.0.2

48. ●Based on Elastic Beanstalk ionroller release SERVICE 0.0.1 3rd Generation (Ionroller) 0.0.1 DNS Application

49. ●Based on Elastic Beanstalk ionroller release SERVICE 0.0.2 3rd Generation (Ionroller) 0.0.1 DNS 0.0.2 Application

50. ●Based on Elastic Beanstalk ionroller release SERVICE 0.0.2 3rd Generation (Ionroller) 0.0.1 DNS 0.0.2 Application

51. ●Based on Elastic Beanstalk ionroller release SERVICE 0.0.2 3rd Generation (Ionroller) 0.0.1 DNS 0.0.2 Application

52. ●Based on Elastic Beanstalk ionroller release SERVICE 0.0.2 3rd Generation (Ionroller) DNS 0.0.2 Application

53. What we didn’t like about those...

54. “We could solve this now, or, just wait six months, and Amazon will provide a solution.” Andrey Kartashov, Distinguished Engineer, Gilt

55. ● Based on CodeDeploy + CloudFormation + AWS CLI (+ ECR) make create stack Development Canary 4th Generation (Mobile Service Tools - MST) Production

56. ● Based on CodeDeploy + CloudFormation + AWS CLI (+ ECR) make publish make deploy-development make deploy-production-canary make deploy-production Development Canary 4th Generation (Mobile Service Tools - MST) Production

57. ● Based on CodeDeploy + CloudFormation + AWS CLI (+ ECR) make deploy-production Development Canary 4th Generation (Mobile Service Tools - MST) Production

58. ● Based on CodeDeploy + CloudFormation + AWS CLI (+ ECR) make deploy-production Development Canary 4th Generation (Mobile Service Tools - MST) Production

59. ●Mostly AWS tooling ●Quick deployments Development Canary 4th Generation (Mobile Service Tools - MST) Production

60. ● Boilerplate scripts ● Docker registry problems ● No cross-account support ● Shared CloudFormation template Development Canary 4th Generation (Mobile Service Tools - MST) Production

61. ● Boilerplate scripts ● Docker registry problems ● No cross-account support ● Shared CloudFormation template 4th Generation (Mobile Service Tools - MST) Base template

62. ●Based on CodeDeploy + CloudFormation + S3 nova stack create ENVIRONMENT Development Canary 5th Generation (Nova) Production Environment

63. ●Based on CodeDeploy + CloudFormation + S3 nova deploy ENVIRONMENT development nova deploy ENVIRONMENT canary nova deploy ENVIRONMENT production Development Canary 5th Generation (Nova) Production Environment

64. ●Versioned CloudFormation templates 5th Generation (Nova) V2V1

65. Happiness

66. FIN, ACK We’ve seen a quick run through today of the benefits of continuous delivery on our software release process: • Continuous integration (build/test) helps shrink our feedback loop greatly • We can get our software out in front of our users much more rapidly • By moving faster we can actually ensure better quality • CodePipeline allows for integration with almost any service or tool you can think of! • Plus visualization of what’s going on!

67. Resources to learn more: • DevOps: https://aws.amazon.com/devops/ • Continuous integration: https://aws.amazon.com/devops/continuous-integration/ • Continuous delivery: https://aws.amazon.com/devops/continuous-delivery/ • CodePipeline • https://aws.amazon.com/codepipeline/ • https://aws.amazon.com/documentation/codepipeline/ • CodeDeploy • https://aws.amazon.com/codedeploy/ • https://aws.amazon.com/documentation/codedeploy/ • https://github.com/awslabs/aws-codedeploy-samples • Code Services Starter Kit: http://bit.ly/AWSCodeStarterKit But wait, there’s more!

68. Please remember to rate this session under My Agenda on awssummit.london

69. Thank you!

Add a comment

Related pages

DevOps on AWS: Deep Dive on Continuous Delivery and the ...

DevOps on AWS: Deep Dive on Continuous Delivery and the AWS Developer Tools
Read more

DevOps on AWS: Deep Dive on Continuous Delivery and the ...

DevOps on AWS: Deep Dive on Continuous Delivery and the ... These services were inspired by Amazon’s own internal developer tools and DevOps ...
Read more

[レポート]DevOps on AWS: Deep Dive on Continuous Delivery and ...

... DevOps on AWS: Deep Dive on Continuous Delivery and the AWS Developer Tools #AWSSummit. ... Android Developer Tools;
Read more

AWS Summits 2016 | Stockholm | Sessions

Developer Tools. AWS CodeCommit. AWS ... DevOps on AWS: Deep Dive on Continuous Delivery and the AWS Developer Tools Customer Speaker: Alma Talent: IoT, ...
Read more

Devops Aws | LinkedIn

View 684 Devops Aws ... is best accomplished using Amazon proprietary administrative tools. The DevOps ... DevOps on AWS: Deep Dive on Continuous Delivery ...
Read more

Devops | LinkedIn

... when I have asked this myself I always get a list of tools back that enable DevOps ... DevOps (Developer ... AWS: Deep Dive on Continuous Delivery ...
Read more

DevOps in AWS Radio Podcast: Episode 001 - stelligent.com

CI in AWS; Resources. DevOps in AWS Solutions; Tools ... recent DevOps in AWS news and do a deep dive into ... such as Continuous Delivery in the ...
Read more

AWS re:Invent 2014 | (APP308) Chef on AWS: Deep Dive - YouTube

AWS re:Invent 2014 | (APP308) Chef on AWS: Deep Dive ... Continuous Delivery and Pipeline ... Elastic Load Balancing Deep Dive and Best ...
Read more