Wix Architecture at Scale - QCon London 2014

50 %
50 %
Information about Wix Architecture at Scale - QCon London 2014

Published on March 7, 2014

Author: aviranwix

Source: slideshare.net


In this talk I will go over Wix's architecture, how we evolved our system to be highly available even at the worst case scenarios when everything can break, how we built a self-healing eventual consistency system for website data distribution and will show some of the patterns we use that helps us render 45M websites while maintaining a relatively low number of servers.

Wix Architecture at Scale Aviran Mordo Head of Back-End Engineering @ Wix @aviranm linkedin.com/in/aviran aviransplace.com

Wix in Numbers Over 45,000,000 users 1M new users/month Static storage is >800TB of data 1.5TB new files/day 3 data centers + 2 clouds (Google, Amazon) 300 servers 700M HTTP requests/day 600 people work at Wix, of which ~ 200 in R&D

Initial Architecture Tomcat, Hibernate, custom web framework Built for fast development Stateful login (Tomcat session), Ehcache, file uploads No consideration for performance, scalability and testing Intended for short-term use Lighttpd (file serving) Wix (Tomcat) MySQL DB

The Monolithic Giant One monolithic server that handled everything Dependency between features Changes in unrelated areas of the system caused deployment of the whole system Failure in unrelated areas will cause system wide downtime

Breaking the System Apart

Concerns and SLA Edit websites Data Validation Security / Authentication Data consistency Lots of data View sites, created by Wix editor High availability Serving Media High availability High performance High performance High traffic volume Lots of static files Long tail Very high traffic volume Viewport optimization Cacheable data

Wix Segmentation Networking 1. Editor Segment 2. Media Segment 3. Public Segment

Making SOA Guidelines Each service has its own database (if one is needed) Only one service can write to a specific DB There may be additional read-only services that directly accesses the DB (for performance reasons) Services are stateless No DB transactions Cache is not a building block, but an optimization

1. Editor Segment

Editor Server Immutable JSON pages (~2.5M / day) Site revisions Active – standby MySQL cross datacenters Editor Server MySQL Active Sites MySQL Archive

Protect The Data Protect against DB outage with fast recovery = replication Protect against data poisoning/corruption = revisions / backup Make the data available at all times = data distribution to multiple locations / providers

Saving Editor Data Browser Save Page(s) 200 OK Editor Server Upload Static Grid Notify Save Page MySQL Active Sites MySQL Archive DC replication MySQL Active Sites MySQL Archive Download Page Notify Archive (Google) Google Cloud Storage Archive (Amazon)

Self Healing Process Browser Save Page(s) 200 OK Editor Server Upload Static Grid Notify Save Page MySQL Active Sites MySQL Archive DC replication MySQL Active Sites MySQL Archive Download Page Notify Archive (Google) Google Cloud Storage Archive (Amazon)

No DB Transactions Save each page (JSON) as an atomic operation Page ID is a content based hash (immutable/idempotent) Finalize transaction by sending site header (list of pages) Can generate orphaned pages, not a problem in practice

2. Media Segment

Prospero – Wix Media Storage 800TB user media files 3M files uploaded daily 500M metadata records Dynamic media processing • Picture resize, crop and sharpen “on the fly” • Watermark • Audio format conversion

Prospero Eventual consistent distributed file system Multi datacenter aware Automatic fallback cross DC Run on commodity servers & cloud

Prospero – Wix Media Manager Tampa Google Cloud x36 x36 T x32 T Second fallback First fallback Austin CDN get image.jpg If not in CDN x36 x36 T x32 T

3. Public Segment

Public Segment Roles Routing (resolve URLs) www.example.com HTML Renderer Dispatching (to a renderer) HTML SEO Renderer Flash SEO Renderer Flash Renderer Sitemap Renderer Robots.txt Renderer Rendering (HTML,XML,TXT) Public Server

Public SLA Response time <100ms at peak traffic

Publish A Site Publish site header (a map of pages for a site) Publish routing table Publish site header / routes Editor Segment Public Segment

Built For Speed Minimize out-of-service hops (2 DB, 1 RPC) Lookup tables are cached in memory, updated every 5 minutes Denormalized data – optimize for read by primary key (MySQL) Minimize business logic

How a Page Gets Rendered Bootstrap HTML template that contains only data Only JavaScript imports JSON data (site-header + dynamic data) No “real” HTML view

Offload rendering work to the browser

The average Intel Core i750 can push up to 7 GFLOPS without overclocking

Why JSON? Easy to parse in JavaScript and Java/Scala Fairly compact text format Highly compressible (5:1 even for small payloads) Easy to fix rendering bugs (just deploy a new client code)

Minimum Number of Public Servers Needed to Serve 45M Sites 4

Public SLA Be Available 99.99999%

Serving a Site – Sunny Day Browser http://example.wix.com HTTP Request HTML Resources / Media CDN Notify site view HTTP Request LB Store HTML to cache Public Rendere r Archiv e Statics

Serving a Site – DC Lost Browser http://example.wix.com CDN Statics HTTP Request LB Archiv e LB Public Public Rendere r Rendere r Change DNS

Serving a Site – Public Lost Browser http://example.wix.com HTTP Request CDN HTML Get Cached HTML Version LB Public Rendere r Archiv e Statics

Living in the Browser Browser http://example.wix.com HTTP Request Fallback Editor LB Public Rendere r HTML JSON / Media CDN Fallback Archiv e Statics

Summary Identify your critical path and concerns Build redundancy in critical path (for availability) De-normalize data (for performance) Minimize out-of-process hops (for performance) Take advantage of client’s CPU power

Q&A http://goo.gl/Oo3lGr Aviran Mordo Head of Back-End Engineering @ Wix @aviranm linkedin.com/in/aviran aviransplace.com

Add a comment

Related presentations

Presentación que realice en el Evento Nacional de Gobierno Abierto, realizado los ...

In this presentation we will describe our experience developing with a highly dyna...

Presentation to the LITA Forum 7th November 2014 Albuquerque, NM

Un recorrido por los cambios que nos generará el wearabletech en el futuro

Um paralelo entre as novidades & mercado em Wearable Computing e Tecnologias Assis...

Microsoft finally joins the smartwatch and fitness tracker game by introducing the...

Related pages

Wix Architecture at Scale - InfoQ: Software Development ...

Wix Architecture at Scale. ... Aviran Mordo introduces Wix's architecture, ... QCon London 2014; Availability; Consistency;
Read more

Speaking @QCon London 2014 (Get A Coupon) | Aviran’s Place

Today I have a special treat for all the readers. I will be speaking at QCon London 2014 about Wix’s architecture at scale, which I highly recommend (-:
Read more

QCon London 2014, 3rd-7th March 2014 | Lanyrd

QCon London 2014 International ... Wix Architecture at Scale. For Slides. Video of Wix Architecture at Scale; ... QCon London @qconlondon; Trifork @ ...
Read more

QCon London 2014 Conference

See you next year at QCon London 2015, ... Architecture Improvements: ... See the QCon London 2014 video calendar here.
Read more

QCon London 2016 | Software Development Conference

Software is changing the world. QCon empowers software development by facilitating the spread of knowledge and innovation in the developer community.
Read more

QCon London 2014 - InfoQ: Software Development News ...

QCon London 2014 Conference Videos. To ... The publishing schedule can be found on the QCon London web ... Continuous Deployment Avleen Vig Wix ...
Read more

Wix | LinkedIn

Wix Architecture at Scale - QCon London 2014. 29,784 Views. nirkoso. UX @ Wix, keys for a successful UX design process . 2,089 Views. efratao. Wix ...
Read more