Published on February 19, 2014
Distributed Reactive Architecture with Rabbit MQ Rabbit MQ Steve Pember Today, 2014
A Few Questions
Have you ever... Wondered why your company’s codebase is all in one repo? codebase is all in one repo?
Have you ever... thought “why are there so many #@$!% tables in this database?
Have you ever... Dreaded executing the unit test suite?
Have you ever... Gone through a major refactoring of your app?
Have you ever... Gone on an ‘archeological expedition’ on the codebase just to add a new feature?
With “Traditional” Monolithic Architecture, One can Reach MVP quickly.
The entirety of the application’s functionality is in one convenient location.
But that’s it.
WAIT! Won’t scale?
...Sure, but that’s not what I mean.
Architecture choice is more valuable than any framework
... which creates faster, leaner code ...
... which results in rapid long-term development time...
... and easier code maintainability...
... which saves you Money.
Also, it’s fun! l
Each node can be written without the methods and technologies best suited to it
Separation of Concerns is a Good Thing.
Following this approach requires an architect to drive the distributed vision and keep folks in line. Y
There’s some non-trivial upfront time investing in a service communication format.
Intra-Service communication has a cost, particularly if it’s synchronous.
Event-Driven Scalable Resilient Responsive
“An application must be reactive from top to bottom.” -The Manifesto
Drop synchronous HTTP. Embrace Events.
Route Messages asynchronously through a message queue when data needs processing
RESTful Messages and Events... ... Consumers only operate on data in the Message.
Add additional nodes to handle load without additional configuration
Plus Additional Decoupling on…
But which Broker to pick?
Rabbit MQ Language Agnostic (additional decoupling) Message Persistence Message Recovery
Slower than other Brokers due to the amazing features.
Especially with Persistence
Leaping* Into Rabbit MQ *I’m sorry.
Work Queues will likely be your default pattern
The Biggest Case Study
Change was Needed
Monolithic: 200-300 requests/ sec /host Reactive: 10 - 20K requests / sec / host
During the 2010 World Cup, Twitter hit a record of 3283 TPS
Average 5700 TPS Dynamically scale to 143,199 TPS…
With 5x - 12x fewer machines than before
Image Credits Complexity (pipes): http://www.flickr.com/photos/bhophoto/384574407/ Simple Pipeline: http://tierracos.com/our-companies/tierra-pipeline/ Highway: http://farnorthdallas.advocatemag.com/wp-content/uploads/2013/04/882349_500402383328407_539732406_o.jpg Homer & doughnuts: http://thechurchillreview.blogspot.com/2012/10/feeling-terror-too-young-aka-kiddie.html Tom Brady: http://stamperdad.wordpress.com/category/tom-brady/ Telephone Exchange Operator: http://fineartamerica.com/featured/telephone-exchange-1920s-granger.html People in Queue: http://www.guardian.co.uk/money/2010/mar/23/dole-queue-jobseekers-online Cookie Monster: http://muppet.wikia.com/wiki/Cookie_Monster_Through_the_Years Scrooge McDuck: http://smallbusiness.uprinting.com/how-pennies-are-hurting-small-business/ Too Many Cooks: http://www.ecommercesystems.com/featured-articles/cooks-kitchen-driving-ecommerce-business/ Clustered Macs: http://www.uiowa.edu/mihpclab/micg.html Resuscitate: http://www.dailymail.co.uk/health/article-2034160/Do-resuscitate-Theyre-fateful-words-meaning-doctors-wont-try-save-you-collapse-hospital.h Iron Man: http://images.wikia.com/marvelmovies Mail Sorting: http://riversidechamber.files.wordpress.com Twitter Logo & app chart: https://blog.twitter.com/engineering Modern Server Farm: http://bookriot.com/2013/03/26/book-less-libraries-and-other-contemporary-realities/ Grey World: http://upload.wikimedia.org/wikipedia/commons/d/d0/BlankMap-World-1ce.png
Although reactive architecture is ... apply a reactive architecture. A refresher: Reactive ... Distributed Reactive Architecture: Extending SOA with Events.
support distributed applications, ... models based on events, or 3) reactive models that allow to the server ... Extending the interaction model between ...
... for building distributed, reactive ... found in the Reactive Microservices Architecture ... can be done by extending ...
Distributed Reactive ... Implementing Federated Object Systems Tobias Freudenreich, ... On-Demand ELT Architecture for Right-Time BI: Extending the ...
Extending the service oriented architecture ... decision parameters in their respective events during ... Oriented Architecture (SOA), ...
A SOA approach to delay and jitter ... both w.r.t. to accurate time-stamping of physical events in a distributed ... Extending the web architecture to this ...
An event-based solution that uses events to ... In Third Intl. Workshop on Distributed ... Reference model for service oriented architecture 1.0 ...