How Apache Drives Music Recommendations At Spotify

50 %
50 %
Information about How Apache Drives Music Recommendations At Spotify

Published on September 29, 2015

Author: JoshBaer

Source: slideshare.net

1. HowApache Drives Music Recommendations At Spotify Josh Baer (jbx@spotify.com) Note:The view expressed is my own and does not necessarily represent that of Spotify

2. WhoAm I? • Technical Product Owner at Spotify • Working with batch and fast processing infrastructure @l_phant

3. Music Discovery in the 90s

4. What is Spotify? • Music Streaming Service • Launched in 2008 • Free and PremiumTiers • Available in 58 Countries

5. 75+ MillionActive Users

6. 30+ Million Songs

7. 1+ Billion Plays/Day

8. Music Recommendations with Apache

9. How do we recommend a personalized playlist of new music to 75+ million users?

10. 10.123.133.333 - - [Mon, 3 June 2015 11:31:33 GMT] "GET /api/admin/job/aggregator/status HTTP/1.1" 200 1847 "https://my.analytics.app/ admin" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36" 10.123.133.222 - - [Mon, 3 June 2015 11:31:43 GMT] "GET /api/admin/job/aggregator/status HTTP/1.1" 200 1984 "https://my.analytics.app/ admin" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36” 10.123.133.222 - - [Mon, 3 June 2015 11:33:02 GMT] "GET /dashboard/courses/1291726 HTTP/1.1" 304 - "https://my.analytics.app/admin" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36" 10.321.145.111 - - [Mon, 3 June 2015 11:33:03 GMT] "GET /api/loggedInUser HTTP/1.1" 304 - "https://my.analytics.app/dashboard/courses/ 1291726" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36" 10.112.322.111 - - [Mon, 3 June 2015 11:33:03 GMT] "POST /api/instrumentation/events/new HTTP/1.1" 200 2 "https://my.analytics.app/ dashboard/courses/1291726" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36” 10.123.133.222 - - [Mon, 3 June 2015 11:33:02 GMT] "GET /dashboard/courses/1291726 HTTP/1.1" 304 - "https://my.analytics.app/admin" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36" It begins with a log

11. Apache Kafka at Spotify •340 Kafka-related nodes •30TB/day from logs

12. How do we store TBs of new data every data?

13. Apache Hadoop at Spotify • 1700 Nodes • 60 PB of Data • 70TB of Memory • Over 1 Million jobs run in Q3, 2015

14. ProcessingGrowth 150% 250% 350% 450% 550% Q4-2013 Q1-2014 Q2-2014 Q3-2014 Q4-2014 Q1-2015 Q2-2015 Q3-2015 Hadoop at Spotify

15. Processing Toolbox • Apache Crunch • Scalding • Apache Hive • Apache Spark • Apache Storm • Hadoop Streaming • Apache Pig

16. Storage Formats • ApacheAvro • Apache Parquet

17. How do we personalize the playlists?

18. Collaborative Filtering Justin Bieber Drake Avicii Major Lazer Anna Listened Listened Gustav Listened Listened Listened Mary Listened Listened Listened Listened Michael Listened ListenedSuggest

19. How do we serve new playlists to all our users every week?

20. Apache Cassandra at Spotify • Number of Clusters: 113 • Number of Machines: 1155 • Largest Cluster: 60 Nodes

21. Driven By Data

22. Driven ByApache

23. Thank YOU for your contributions to Apache products!

24. One Last Thing…

25. Spotify Luigi •Workflow Manager •Over 150 contributors •Used by 10s, possibly 100s of companies

26. Maybe…Apache Luigi? Sponsors/mentors/contributors wanted!

27. Think this stuff is interesting? We have a great time building it! spotify.com/jobs

28. Better Spotify ML Presentations • Algorithmic Music Recommendations at Spotify (Chris Johnson) • Interactive Recommender Systems with Netflix and Spotify (Chris Johnson) • Music recommendations @ MLConf 2014 (Erik Bernhardsson) • Machine learning @ Spotify (Andy Sloane) • Recommending music on Spotifywith deep learning (Sander Dieleman) • Scala Data Pipelines @ Spotify (Neville Li) • Spotify's Music Recommendations LambdaArchitecture (Esh Kumar and Emily Samuels)

Add a comment

Comments

 | 21/11/15
 | 04/12/15
 | 04/12/15
 | 05/12/15
 | 06/12/15
 | 06/12/15
h1vw6cq161 | 23/01/16
http://www.msnbrand.com/goods-copy-4704.html スーパーコピー時計の激安老舗.!国内No.1時計コピー工房,アフターサービスも自ら製造したスーパーコピー時計なので、技術力でお客様に安心のサポー トをご提供させて頂きます。スーパーコピー 代引きN品をご 購入の方は、こちらへ.弊社は正規品と同等品質のコピー品を低価で お客様に提供します!すべての商品は品質2年無料保証です。 }}}}}}
r8ip6sj685 | 05/02/16
http://www.brandiwc.com/Super-Copy-27.html スーパーコピーブランド弊社は安心と信頼のスーパーコピーブランド (N級品)専門店です!全国送料無料!日本一流品質のスーパーコピー時計、ブランド財布コピー、ブランドバッグコピー新作最新入荷!ロレックススーパーコピー,ウブロ スーパーコピー,ブランド時計 コピー,ブランド スーパーコピー,コピーブランド 通販,スーパーコピー 財布その他の世界一流ブランドコピーを取り扱っています。 }}}}}}
 | 29/08/16
 | 28/09/17
 | 01/10/17
N

Related pages

Keynote: How Apache Drives Music Recommendations - YouTube

Hear from Josh Baer, Hadoop Product Owner at Spotify on how Apache drives Spotify's music recommendations. Speakers Josh Baer Hadoop Product ...
Read more

SF Scala: Nikhil Tibrewal, Playlist Recommendations ...

Music recommendations on Spotify is a ... SF Scala: Nikhil Tibrewal, Playlist Recommendations ... How Apache Drives Music Recommendations ...
Read more

Apache Conferences, BarCamps, and MeetUps

Josh Baer, Hadoop lead at Spotify, will share how Apache drives Spotify’s music recommendations. Dirk Duellmann, deputy leader of the data and storage ...
Read more

Apache Big Data Budapest, Sep 28-30 – use code “ABDKDN25 ...

Apache Big Data Budapest, ... Apache: Big Data keynote ... Hadoop lead at Spotify, will share how Apache drives Spotify's music recommendations.
Read more

Music for everyone - Spotify

Spotify is a digital music service that gives you access to millions of songs. ... There are millions of songs on Spotify. Play your favorites, ...
Read more

Music Recommendations at Scale with Spark - Spark Summit

Music Recommendations at Scale with Spark. ... Spotify uses a range of Machine Learning models to power its music recommendation features including the ...
Read more

Partner Finance & Strategy Manager – New York, NY - Spotify

... we’re focused on providing the right music ... You will perform deal forensic analysis and develop actionable recommendations; ... You will drive ...
Read more