Big Data Universe - How we design architectures

50 %
50 %
Information about Big Data Universe - How we design architectures

Published on May 31, 2016

Author: gulyasm

Source: slideshare.net

1. How we design data architecture Mate Gulyas

2. CTO & Co-Founder GULYÁS MÁTÉ @gulyasm

3. ARCHITECTURE? ● CODE ARCHITECTURE ● GENERAL INFRASTRUCTURE ● DATA INFRASTRUCTURE @gulyasm

4. ON THE NEXT EPISODE OF BIG DATA... 1.WHAT DO WE DESIGN FOR? 1.OUR STORY, OUR FAILURES @gulyasm

5. WHAT DO WE DESIGN FOR?

6. WHAT DO WE DESIGN FOR? ● SCALABILITY ● MAINTAINABILITY ● COST @gulyasm

7. SCALABILITY AND MAINTAINABILITY ARE RESULTS OF A GOOD DESIGN

8. WHAT DO WE REALLY DESIGN FOR? ● SIMPLICITY ● RESILIENCY ● SMALL ITERATIONS ● SELF SERVICE @gulyasm

9. WHAT DO WE REALLY DESIGN FOR? ● SIMPLICITY ● RESILIENCY ● SMALL ITERATIONS ● SELF SERVICE @gulyasm

10. SIMPLICITY SIMPLE THINGS SCALE WELL @gulyasm

11. SIMPLICITY SIMPLE THINGS ARE EASY TO UNDERSTAND @gulyasm

12. SIMPLICITY BORING TECHNOLOGY IS GOOD TECHNOLOGY @gulyasm

13. SMALL ITERATIONS THE UNKNOWNS ● THE UNKNOWNS ● THE UNKNOWN UNKNOWNS @gulyasm

14. SMALL ITERATIONS @gulyasm

15. END RESULT @gulyasm

16. SMALL ITERATIONS @gulyasm

17. SMALL ITERATIONS @gulyasm

18. SMALL ITERATIONS @gulyasm

19. SMALL ITERATIONS @gulyasm

20. SMALL ITERATIONS @gulyasm

21. SMALL ITERATIONS @gulyasm

22. SELF SERVICE YOUR SOFTWARE/IT INFRASTRUCTURE IMPACTS THE WHOLE ORGANIZATION

23. ENBRITELY DATA PLATFORM

24. Product placeholder

25. Luigi TOOLS Luigi + enbrite.ly extensions = Gabo Luigi WORKFLOW ENGINE

26. Tools we created GABO LUIGI

27. Spark TOOLS 0.5-4TB daily data 1-10B events Ad-hoc batch queries: 20TB data

28. Spark TOOLS ● SPENT 3 MONTHS OPTIMIZING IT ● 20+ NODE CLUSTERS ● UNIT TESTS

29. AWS TOOLS ● 16 services ● 110+ machines ● 1-4 EMR clusters (1-20 node) ● 100TB+ on S3 ● All clients has separate infrastructure

30. HOW WE GOT HERE? MONOLITHIC PYTHON ANALYTICS EVALUATE BIG DATA TECHNOLOGIES STARTED WORK ON DP DP PRODUCTION READY SAAS DP @gulyasm

31. HAVE FUN! @gulyasm

32. PRACTICE AT HOME @gulyasm

33. WE ARE HIRING!

34. WE ARE HIRING!

35. MATE GULYAS gulyasm@enbrite.ly @gulyasm @enbritely THANK YOU!

Add a comment

Related presentations

Related pages

Big Data Meets Big Data Analytics - SAS

Big Data Meets Big Data ... Difficulties dealing with data increase with the expanding universe of data ... Big Data Analytics: Future Architectures, ...
Read more

Real-Time Big Data Analytics: Emerging Architecture

Change the world with data. We’ll show you how. ... A rapidly emerging universe of newer technologies has dramatically ... real-time big data analytics ...
Read more

IDC I V I E W Extracting Value from Chaos - emc.com

New storage management tools are available to cut the costs of the part of the digital universe we ... design. Each step will ... Big data is a horizontal ...
Read more

Big Data in the Enterprise - Network Design Considerations ...

Big Data in the Enterprise: Network Design ... This document also shows that the Cisco Nexus® architectures are optimized to handle big ... Big data is a ...
Read more

Analytics, Big Data, and Data Science Courses - Big Data ...

Free big data and data science courses and learning paths. ... We have courses for all skill levels
Read more

Indexing and Processing Big Data - Page d'accueil / Lirmm ...

Indexing and Processing Big Data ... • And we can now store these data! ... Digital Universe study of International Data
Read more

Towards a Big Data Reference Architecture

Towards a Big Data Reference ... these new characteristics of data, design a reference ... to describe how big data transforms the way ‘we ...
Read more

Universe Design | LinkedIn

View 4793 Universe Design posts, presentations, experts, and more. ... Data Warehousing (450 members) PL/SQL (435 members) Shell Scripting (407 members)
Read more