Node in Production at Aviary

50 %
50 %
Information about Node in Production at Aviary

Published on March 6, 2014

Author: aviary



Aviary's customizable SDK powers cross-platform photo editing for over 6,500 partners and over 70 million monthly active users across the globe. Some of our notable partners include Walgreens, Squarespace, Yahoo Mail, Flickr, Photobucket, and Wix. At Aviary, we use node.js for several mission-critical projects in production and have seen extremely positive results. In this talk, we will discuss how we approach some common situations that developers deploying node.js projects will likely need to tackle. We will walk you through our routing mechanism, our automated deployment system, some of our custom middleware, and our testing philosophy.

Node.js in Production at Aviary NYC Node.js Meetup March 5, 2014

Aviary Photo-Editing SDK & Apps Fully-Baked UI Configurable, High-Quality Tools Over 6,500 Partners Over 70 Million Monthly Users Over 6 Billion Photos Edited iOS, Android, Web, Server J

Who Are We? Nir Jack Lead Serverside Director of Engineer Engineering Likes: ● ● ● Automated deployment Big-O notation Brainteasers Hates: ● Cilantro Likes: ● ● ● Parallelizing processes DRY code Seltzer Hates: ● Food after the sell-by date

Who Are We? Jeff Ari Serverside Developer Engineer Evangelist Likes: ● ● ● Performance Profiling Spaces, not tabs Bikes Hates: ● His Photo Likes: ● ● ● Empowering Developers Refactoring/Patterns Dancing Hates: ● Forrest Gump

How Do We Use Node? ● In Production: ○ ○ ○ ○ ○ ● Future: Analytics Dashboard ○ Server-side Rendering API Content Delivery (CDS) ○ Automated billing Public Website Receipts Collection Monitoring Tools

Why Do We Use Node? ● ● ● ● ● ● ● ● Extremely fast and lightweight Easy to iterate on Common language for client and server JSON Cross Platform npm express module Active Community

Setting Up Your Server

Request Routing Our API servers all require(routes.json) { “home”: { “getVersion”: { “verb”: “get”, “path”: “/version” } }, “queue”: { “updateContent”: { “verb”: ”put”, “path”: “/content/:id”, “permissions”: [“content”] } } } for (var controllerName in routes) { var controller = require(ctrlrDir + controllerName); for (var endpointName in routes[controllerName]) { var endpoint = routes[controllerName][endpointName]; var callback = controller[endpointName]; app[endpoint.verb]( endpoint.path, ensurePermissions(endpoint.permissions), callback ); } }

Authentication: Overview Request Server listens Middleware Logged in? No Yes Response Request handler takes over Does user have permission? Redirected to login page Authenticated user saved in cookie

Authentication: Login passport.use(new GoogleStrategy({ returnURL: DOMAIN + '/auth/return' }, function (identifier, profile, done) { var userInfo = { name:, fullName: }; userRepository.findUserByName(, function (findErr, foundUser) { // ... if (foundUser.length === 0) { done('Invalid user', null); return; } userInfo.userId = foundUser.user_id; userInfo.permissions = foundUser.permissions; done(null, userInfo); }); } ));

Working with JSON

Validation - JSON Schema SCHEMA ● JSON-based JSON { { “type”: “object” “additionalProperties”: false “properties”: { “status”: { “type”: “string”, “enum”: [“ok”, “error”], “required”: true }, “data”: { “type”: “object”, “required”: false } } ● Like XML Schema ● Validation modules ● Used throughout Aviary’s systems } “status”: “ok” } { “status”: “error”, “data”: { “reason”: “hoisting” } } { “status”: “gladys”, “node”: “meetup” }

Advanced JSON - Content Effects Frames Stickers Messages

The One-To-Many Problem Android expects responses to look like this: iOS 1.0 expects responses to look like this: iOS 2.0 expects responses to look like this:

Response Formatting - The Model Content Entry Response Formats Responses JSON document describing content item JSON documents defining mappings from entry to responses Actual JSON responses delivered to devices

Response Formatting - Details The Single Content Entry "identifier": "com.aviary.stickers.234fe" The Response Format "id": { "dataKey": "identifier" "icon": { "path": "cds/hats/icon.png" "path-100": "cds/hats/icon100.png" }, "isPaid": { "isPaid": true, "iconImagePath": "cds/hats/icon100.png" "stickers": [ { "type": "boolean", }, "dataKey": "isFree", { "identifier": "1" "transformations": ["negateBool"] "items": [ "imageUrl": "cds/hats/1.png" }, "identifier": "1" } "iconImagePath": { "imageUrl": "cds/hats/1.png" "type": "string", } ] "id": "com.aviary.stickers.234fe", "type": "string", "isFree": false, The Response "dataKey": "icon.path-100" }, "stickers": { "type": "array", "dataKey": "items" ]

Code Sample (Dumbed Down) var formattedResponse = {}; for (var propName in responseFormat) { var val = contentEntry[responseFormat[propName].dataKey]; for (var transformation in responseFormat[propName].transformations) { val = transformationModule[transformation](val); } formattedResponse[propName] = val; } return formattedResponse;

Interacting with External Processes

Image Rendering Challenge: Use our existing image rendering .NET/C++ process from node server Solution: require(‘child_process’).spawn(‘renderer.exe’) Benefits: Easy IPC, asynchronous workflow

Code Sample var spawn = require(‘child_process’).spawn; var renderer = spawn(‘renderer.exe’, [‘-i’, ‘inputImage.jpg’, … ]); // read text renderer.stderr.setEncoding(‘utf8’); renderer.stderr.on(‘data’, function (data) { json += data; }); // or binary data renderer.stdout.on(‘data’, function (data) { buffers.push(data); }); renderer.on(‘close’, function (code, signal) { // respond to exit code, signal (e.g. ‘SIGTERM’), process output var diagnostics = JSON.parse(json); var img = Buffer.concat(buffers); });

Going Live

Testing Philosophy ● Unit tests (sparingly) ● End-to-end integration tests ● Mocha ● Enforced before push ○ (master / development)

Example Integration Test #!/bin/bash ● Bash script mocha scopecreation && mocha cmsformatcreation && mocha crfcreation && ● Independent files mocha mrfcreation && mocha rflcreation && mocha appcreation && ● Shared configuration mocha contentcreation && mocha manifestcreation && mocha push && ● Single failure stops process mocha cmsformatupdate && mocha crfaddition && mocha rfladdition && mocha contentupdate && mocha manifestupdate

Automated Deployment: Overview Git S3 2) Jenkins polls for repo changes 1) Code is pushed to master 3) Code is zipped and uploaded to S3 Jenkins 4) Get a list of live servers in this group AWS API 5) SSH into each server and run the bootstrap script

Automated Deployment: Bootstrap 5) SSH into each server and run the bootstrap script #!/bin/bash ZIP_LOCATION="s3://aviary/projectX/"; cd ~/projectX; sudo apt-get -y -q install nodejs@0.10.0; sudo apt-get -y -q install s3cmd; sudo npm install -g forever@0.10.8; Goals of the 1. Ensure all dependencies are installed 2. Download and extract project 3. Ensure HTTP traffic is routed to the proper port 4. Keep the old version of the project live until the moment the new one is ready to go live # Missing step: create s3 configuration file s3cmd -c /usr/local/s3cfg.config get "$ZIP_LOCATION"; unzip -q -o iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-ports 8142; forever stopall; forever start server.js;


Lessons Learned (1) ● Integration tests! ● Watch out for node and npm updates ○ Hardcode the node version you’re using ○ If you’re using package.json, version everything ● Node.js + MongoDb are a great couple ● Make sure you understand hoisting

Lessons Learned (2) ● Always callback in async functions ● Always return after a callback ● Node doesn’t always run the same on all platforms ● Use middleware only when necessary ● Always store dates as Unix Timestamps ○ Timezones are a pain in your future ● Throwing unhandled errors will crash your process

Conclusion Today, our production node servers: ● serve dynamic content to 20MM people (soon 70MM) ● power our website: ● log real-time receipt data for every in-app purchase ● allow us to analyze hundreds of millions of events daily ● power quick scripts and one-off internal tools

Questions? Comments also welcome - - - …and by the way, WE’RE HIRING!

Add a comment

Related presentations

Related pages

Winter Wednesdays at Tracy Aviary |

Tracy Aviary is offering one dollar admission every Wednesday now through March. ... KUED Local Productions Menu. Independent Producers; Production ...
Read more

Digital compositing - Wikipedia, the free encyclopedia

... is no longer in production). ... Node-based compositing packages often handle keyframing and time effects ... Aviary Peacock; Avid DS; CompTime (ILM)
Read more

eyeon Fusion - Wikipedia, the free encyclopedia

Blackmagic Fusion (formerly known as ... It employs a node-based interface in which complex processes are built up by connecting a ... Aviary Peacock; Avid ...
Read more

Martin Kastner: Aviary's glass master - Time Out • The ...

... [node:148751 link=Alinea;] ... Kastner’s team’s answer (he works with two production coordinators): ... The Aviary | Cocktail bar ...
Read more

Aviary Advanced Suite Alternatives -

Popular Alternatives to Aviary Advanced Suite. ... Nodewerk – Node based visual ... Dec 2012 Suggestion to add Memeration as an alternative to Aviary was ...
Read more