Meson: Heterogeneous Workflows with Spark at Netflix

100 %
0 %
Information about Meson: Heterogeneous Workflows with Spark at Netflix

Published on June 8, 2016

Author: aasamy

Source: slideshare.net

1. Heterogeneous Workflows with Spark at Netflix 0 Antony Arokiasamy | Kedar Sadekar | Personalization Infrastructure

2. 1 Help members find content to watch and enjoy to maximize member satisfaction and retention

3. Everything is a Recommendation 2 Recommendations are driven by Machine Learning Ranking Rows

4. Machine Learning Pipeline 3 User Selection Feature Generation Model Validation Publish Model Model Training

5. Machine Learning Pipeline Challenges 4 • Innovation • Heterogeneous Environments • Spark • Native Support • Separate Orchestration and Execution • Multi Tenancy • Machine Learning Constructs • Parameter Sweep – 30k Dockers

6. Meson Workflow System 5 • General Purpose Workflow Orchestration and Scheduling framework • Delegates execution to resource managers like Mesos • Optimized for Machine Learning Pipelines and Visualization • Checkout the Blog • http://bit.ly/mesonws or techblog.netflix.com • Plan to Open Sourced soon

7. Meson Architecture 6

8. Standard and Custom Step Types 7

9. Parameter Passing 8 Hive Query User DataSet Regional DataSet Global DataSet Get Users Regional Model Global Model User DataSet Wrangle Data

10. Structured Constructs 9

11. Top Down or Bottom Up 10

12. Two Way Communication 11

13. Spark Step 12

14. Artifacts 13 • Step outputs tracked as Artifacts • Visualization • Memoization

15. Multi Tenancy 14 • Resource Attributes • spark.cores.max • spark.executor.memory • spark.mesos.constraints • Dynamic Resource Allocation

16. Cluster Management 15 • Red-Black software updates • Scale up/Scale down

17. Meson/Spark Cluster 16 • 100s of Concurrent Jobs • 700 Nodes • 5000 Cores • 25 TB Memory • Apps: Meson Workflow System, Spark and Dockers • Few smaller clusters

18. 17 Antony Arokiasamy Kedar Sadekar @aasamy /aasamy aarokiasamy@netflix.com @kedar_sadekar /kedar-sadekar ksadekar@netflix.com

Add a comment

Related presentations

Related pages

Meson: Workflow Orchestration for Netflix Recommendations

Meson: Workflow Orchestration for Netflix ... workloads across heterogeneous ... back to the Meson scheduler. Spark jobs submitted ...
Read more

The Netflix Tech Blog: Meson: Workflow Orchestration for ...

Meson: Workflow Orchestration for Netflix ... that execute workloads across heterogeneous ... Spark Submit within Meson allows for ...
Read more

Heterogeneous Workflows With Spark At Netflix | Schedule ...

Heterogeneous Workflows With Spark At Netflix. ... Docker and other toolkits into a scalable orchestration framework that supports heterogeneous workloads.
Read more

Antony Arokiasamy - Engineer , Netflix | Speakers | Spark ...

Antony Arokiasamy Engineer , Netflix. ... Heterogeneous Workflows With Spark At Netflix. Tuesday, June 7; ... Apache Spark, Spark, the Spark ...
Read more

The Netflix Tech Blog

... workloads across heterogeneous ... various workflows. Meson delegates ... Spark Submit Step, Hive Query Step, Netflix specific ...
Read more

Simplifying Sorting with Spark DataFrames | Knoldus

5 Responses to Simplifying Sorting with Spark DataFrames. ... #GoneScala Meson: Workflow Orchestration for Netflix Recommendations: At Netflix, ...
Read more

Tuning apache spark application with speculation | Knoldus

3 Responses to Tuning apache spark application with speculation. ... #GoneScala Meson: Workflow Orchestration for Netflix Recommendations: At Netflix, ...
Read more

Amazon EMR Release 4.1.0 – Spark 1.5.0, Hue 3.7.1, HDFS ...

... HDFS Encryption, Presto, Oozie, Zeppelin, ... distributed SQL query engine designed to query large data sets in one or more heterogeneous ... Netflix ...
Read more

Full Hacker News

Meson: Workflow Orchestration for Netflix ... that execute workloads across heterogeneous ... Spark Submit within Meson allows for ...
Read more