advertisement

Scaling Deployment at Etsy

50 %
50 %
advertisement
Information about Scaling Deployment at Etsy
Technology

Published on October 10, 2013

Author: mrtazz

Source: slideshare.net

Description

How deployment is tackled at Etsy and how we keep it fast and scale it out with a growing engineering team.

Reprised version of the talk given at the Continuous Delivery NYC meetup October 2013.
advertisement

Scaling Deployment at Etsy Daniel Schauenberg dschauenberg@etsy.com @mrtazz Thursday, October 10, 13

Thursday, October 10, 13

August 2013 •1.8 billion page views •5,483,399 items sold •$109.1 million of goods sold •> 30 million members •> 1 million active shops http://www.etsy.com/blog/news/2013/etsy-statistics-august-2013-weather-report/ | Items by RockerDollJewellery, ZulamimiLand, codice, 42Things Thursday, October 10, 13

LAMMP Item by TheBackPackShoppe Thursday, October 10, 13

Item by FrankelPhotos Monolithic App Thursday, October 10, 13

No Branching Item by NurseryWallArt Thursday, October 10, 13

Deploy Frequency Thursday, October 10, 13

First Day Item by flowersandfleurons Thursday, October 10, 13

Thursday, October 10, 13

IRC Thursday, October 10, 13

Thursday, October 10, 13

DeveloperVMs •KVM •Dev version of full Etsy stack •Chef •DevTools Thursday, October 10, 13

Thursday, October 10, 13

Item by codecards Thursday, October 10, 13

% review -r dschauenberg Thursday, October 10, 13

automatically assignedautomatically assigned Thursday, October 10, 13

Try Item by CSSDesign Thursday, October 10, 13

Actually ... Thursday, October 10, 13

The Bobs Item by Signz Thursday, October 10, 13

The Bobs •LXC containers on buildtests •Multiple SSDs •Labels for heavy/any execution •One heavy executor per disk Thursday, October 10, 13

CI/Try •~260 Bobs •Mostly for try •Constant monitoring for slow tests Thursday, October 10, 13

push train Item by decomodwalls Thursday, October 10, 13

#push •IRC channel to organize push trains •Join a train if you want to deploy changes •Schedule is planned via the channel topic •First in the train is the driver (controls the deploy) •Opening hours: 7am - 10pm NYC time Thursday, October 10, 13

#push <prod> kseever* + jameslee | jpaul | DanielConvissor (c) Thursday, October 10, 13

#push bateman* + krunal* + enorris* | tristan (c) + jameslee (c) + jlaster (c) | dawa + corey + sandosh + jklein + magera + seth_home + mpascual + nathan | bateman | russp (c) Thursday, October 10, 13

pushbot •.join •.in •.good •.done Thursday, October 10, 13

pushbot Thursday, October 10, 13

Item by EsalonPhotography Thursday, October 10, 13

Deployinator Thursday, October 10, 13

Thursday, October 10, 13

Thursday, October 10, 13

stale commits Thursday, October 10, 13

version checks buttons disabledbuttons disabled Thursday, October 10, 13

version checks Thursday, October 10, 13

lock down deploys Thursday, October 10, 13

https://github.com/etsy/deployinator Thursday, October 10, 13

Downsides •Deploys not atomic on the request level •Limbo during the time of the local rsync •Common strategy was to split commits into 3 deploys Thursday, October 10, 13

Item by Geographicsart Thursday, October 10, 13

Atomic Deploys Thursday, October 10, 13

Basic Idea Yin Yang Active Docroot Thursday, October 10, 13

Basic Idea Yin Yang Active Docroot rsync Thursday, October 10, 13

Basic Idea Yin Yang Active Docroot Thursday, October 10, 13

Basic Idea Yin Yang Active Docroot Thursday, October 10, 13

Problems •Symlink swap during requests •Code needs to be guaranteed to finish on the docroot it started •Code inclusion mid request Thursday, October 10, 13

etsy/mod_realdoc •Apache post_read_request hook •Whole request works on realpath of docroot •Caches realpath for 2s Thursday, October 10, 13

ini_set('include_path', $_SERVER['DOCUMENT_ROOT'].'/../include'); Thursday, October 10, 13

etsy/incpath •PHP module to set the incpath •Gets docroot from Apache or realpath() itself •Looks for a pattern to replace in include_path •Restores include_path at the end of the request Thursday, October 10, 13

What did we get? •Remove functions and call site in same deploy •No restarts necessary •Opcode caches stay warm for files that don’t change between 2 deploys Thursday, October 10, 13

Things to watch out for •Code that uses full path names to scripts •Atomic symlink swapping with `mv -T` •Realpath caching to not stress the filesystem •Opcode cache needs to fit 2x code size •Only request atomicity Thursday, October 10, 13

The Plateau Item by finandfancy Thursday, October 10, 13

The Plateau •Regular deploys took ~15 mins •Config deploys about half •10am - 6pm => ~ 32 deploys •Long waiting times Thursday, October 10, 13

Item by KlaireWarren Thursday, October 10, 13

Waiting for push queueWaiting for push queue Thursday, October 10, 13

Split The Queues Item by KlaireWarren Thursday, October 10, 13

HELLO SPLIT QUEUESHELLO SPLIT QUEUES Thursday, October 10, 13

Dashboards deploy linesdeploy lines Thursday, October 10, 13

Supergrep Thursday, October 10, 13

Thursday, October 10, 13

Summary •Current setup has scaled to ~150 people •Constantly trying to improve the speed of deployment •Find weak parts in the process and make them more robust/faster •Bring Dev closer to Prod •Not being able to deploy has the same status as the site being down Thursday, October 10, 13

codeascraft.etsy.com www.etsy.com/codeascraft/talks etsy.github.com www.etsy.com/careers Thursday, October 10, 13

Questions? Thursday, October 10, 13

Scaling Deployment at Etsy Daniel Schauenberg dschauenberg@etsy.com @mrtazz Thursday, October 10, 13

Add a comment

Related presentations

Presentación que realice en el Evento Nacional de Gobierno Abierto, realizado los ...

In this presentation we will describe our experience developing with a highly dyna...

Presentation to the LITA Forum 7th November 2014 Albuquerque, NM

Un recorrido por los cambios que nos generará el wearabletech en el futuro

Um paralelo entre as novidades & mercado em Wearable Computing e Tecnologias Assis...

Microsoft finally joins the smartwatch and fitness tracker game by introducing the...

Related pages

Scaling Deployment at Etsy - Hakka Labs

In this talk, Scaling Deployment, Daniel Schauenberg of Etsy talks about the development and deployment infrastructure at Etsy.
Read more

Daniel Schauenberg: Scaling Deployment at Etsy - YouTube

At Etsy we have over 100 engineers deploying more than 40 times a day. This culture of continuously deploying small change sets enables us to ...
Read more

Scaling Deployment at Etsy // Speaker Deck

Scaling Deployment at Etsy by Daniel Schauenberg. Published April 18, 2013 in Technology. How we keep deployment fast at Etsy, presented at Scaleconf 2013.
Read more

Scaling Software Deployment at Etsy by Daniel Schauenberg ...

Scaling Software Deployment at Etsy by Daniel Schauenberg Hakka Labs. ... Daniel Schauenberg: Scaling Deployment at Etsy - Duration: 44:28.
Read more

ScaleConf - Schedule

Continuous deployment to millions of users 40 times a day ... Scaling Infrastructure at Etsy ... this schedule is tentative and subject to ...
Read more

Etsy - Engineering Talks

Etsy Talks Presentations by the people who build and run Etsy. Featured Presentations. ... Scaling Deployment at Etsy (revised) Daniel Schauenberg. October ...
Read more

Scaling CI at Etsy: Divide and Concur, Revisited - Code as ...

Very interesting idea and plugin to divide your projects into logical jobs – Your master jobs are deployment to prod, qa, princess for each of your tabs ...
Read more

Scaling startups – Chad Dickerson's blog

The Scaling Startups panel I was on last week at Supernova generated a little coverage, ... Before Etsy, I had done the ... Scaling a Startup, jQuery ...
Read more