"D" Stands for Disruption: The CDO in the Modern Era

50 %
50 %
Information about "D" Stands for Disruption: The CDO in the Modern Era

Published on October 19, 2016

Author: continuumio

Source: slideshare.net

1. © 2016 Continuum Analytics - Confidential & Proprietary© 2016 Continuum Analytics - Confidential & Proprietary “D” Stands for Disruption:
 The CDO in the Modern Era Peter Wang CTO, Co-founder Continuum Analytics

2. Hello 2 DATA is Everywhere

3. Hello 3 EVERYTHING surrounding the data is changing

4. © 2016 Continuum Analytics - Confidential & Proprietary 4

5. © 2016 Continuum Analytics - Confidential & Proprietary 5 1973 19811968 1974 SQL Numeric 19962005 1993 1991

6. © 2016 Continuum Analytics - Confidential & Proprietary 6

7. 7 THE DATA REVOLUTION is just beginning Technology innovation is accelerating Every aspect of how we ingest, store, manage and compute on business data will be disrupted

8. an inclusive movement that makes open source tools of data science – data, analytics, & computation – easily work together as a connected ecosystem Open Data Science is…

9. Availability | Innovation | Interoperability | Transparency For everyone in the data science team Open Data Science means… OPEN DATA SCIENCE IS THE FOUNDATION TO MODERNIZATION

10. © 2016 Continuum Analytics - Confidential & Proprietary 10 Data Science is not just Machine Learning… Distributed 
 Systems Business 
 Intelligence Machine Learning
 / Statistics Web Scientific 
 Computing / HPC

11. © 2016 Continuum Analytics - Confidential & Proprietary 11 Data Science is Interdisciplinary… Distributed 
 Systems Business 
 Intelligence Machine Learning
 / Statistics Web Scientific 
 Computing / HPC Classification, deep learning, Regression, PCA Hadoop, Spark Web crawling, scraping, 3rd party data & API providers, predictive services & APIs GPUs, multi-coresData warehouse, querying, reporting

12. © 2016 Continuum Analytics - Confidential & Proprietary 12 Numba dask xlwings Airflow BlazeOpen Source Communities Creates Powerful Technology for Data Science Distributed 
 Systems Business 
 Intelligence Web Scientific 
 Computing / HPC Machine Learning
 / Statistics

13. © 2016 Continuum Analytics - Confidential & Proprietary 13 Numba dask xlwings Airflow Blaze Python is 
 the common language Distributed 
 Systems Business 
 Intelligence Web Scientific 
 Computing / HPC Machine Learning
 / Statistics

14. © 2016 Continuum Analytics - Confidential & Proprietary 14 Python’s 
 Not the 
 Only One… Distributed 
 Systems Business 
 Intelligence Web Scientific 
 Computing / HPC SQL Machine Learning
 / Statistics

15. © 2016 Continuum Analytics - Confidential & Proprietary 15 But it’s also a Great Glue Language Distributed 
 Systems Business 
 Intelligence Machine Learning
 / Statistics Web Scientific 
 Computing / HPC SQL

16. © 2016 Continuum Analytics - Confidential & Proprietary 16 Numba dask xlwings Airflow BlazeAnaconda is 
 the Open Data Science Platform bringing technology together… Distributed 
 Systems Business 
 Intelligence Web Scientific 
 Computing / HPC Machine Learning
 / Statistics

17. © 2016 Continuum Analytics - Confidential & Proprietary 17 Open Data Science
 Vibrant and Growing Community Python Community 30M+ Packages in Anaconda 720+ R Community 16M+ Spark Python Usage 60%+ ANACONDA
 Downloads 8M+

18. © 2016 Continuum Analytics - Confidential & Proprietary 18 Open Data Science Platform ACCELERATE. CONNECT. EMPOWER

19. © 2016 Continuum Analytics - Confidential & Proprietary 19 INNOVATE faster through managed agile experimentation MOVE from analysis to deployment immediately DELIVER powerful results backed by high performance open data science platform LEVERAGE innovative open source analytics to extract value from data MAXIMIZE your computational power to easily analyze all data CONNECT and integrate all your data sources for predictive models ITERATE quickly to create powerful analysis and predictive models COLLABORATE and share with your data science team PUBLISH interactive results to 
 the business ACCELERATE Time-to-Value CONNECT Data, Analytics & Compute EMPOWER Data Science Teams

20. The Core Challenge of Open Data Science in the Enterprise 20

21. © 2016 Continuum Analytics - Confidential & Proprietary Common Problems 21 • Data Science Sandbox is on isolated network, “off the reservation” of governance, risk, compliance (GRC) • Provides freedom to data scientists • Protects production ETL, DW, event processing • … but moving anything from Sandbox to Production is a huge pain • Multiple orgs / LOBs interface with Data Science team in the mixed sandbox environment • Compliance, audit, & risk control?

22. © 2016 Continuum Analytics - Confidential & Proprietary Contrasting Concerns 22 Exploration Production Data • Fast, unfettered access • Ease of introducing new, varied, messy datasets • Reproducibility • Strict, governed access • Well-defined schema • Provenance & auditability Compute Infrastructure • High performance • Low latency, interactive • Individualized & specialized • Scalable, high-availability • Manageable at scale • Cost amortization over many machines and users Organization • Individual high-achievers with lots of context & capability • Agile, able to quickly learn new skills and approaches • Sustain operations at lowest possible cost • Robustness against unintended change

23. © 2016 Continuum Analytics - Confidential & Proprietary • Data Exploration generates insight & is required to respond to business challenges • Production data processing & analytics requires different operational concerns • Over-engineering for either leads to structural deficiencies • Modern & future needs will require more agile exploration Core Challenges 23

24. © 2016 Continuum Analytics - Confidential & Proprietary Conway’s Law 24 The design of any piece of software reflects the communications structure of the organization that produced it.

25. © 2016 Continuum Analytics - Confidential & Proprietary Peter’s Corollary to Conway’s Law 25 The architecture of any business data system evolves to reflect the budget structure of the IT groups that maintain it. … not strategic or operational needs … not ensuring future analytical agility … not optimizing for rapid insights

26. © 2016 Continuum Analytics - Confidential & Proprietary • How businesses are used to buying can actually push power away from exploratory data science capabilities • Information systems have ossified into “software & hardware”, which is fine for straightforward data processing • Not suited for human-in-the-loop production of inference, insight, knowledge “Don’t Starve the Unicorns” 26

27. © 2016 Continuum Analytics - Confidential & Proprietary • VERY common misconception • Python is probably the most misunderstood language • There are “tribes” and ecosystems in Python: web dev, scipy, pydata, embedded, scripting, 3D graphics, etc. • But businesses tend to pigeonhole it: • IT/software/data engineering view: competes with Java, C#, Ruby… • Analytics, stats, data science view: competes with R, SAS, Matlab, SPSS, BI systems Data Science != Software Development 27 Python done right can be a powerful, unifying force across the business.

28. Anaconda for Open Data Science in Hadoop 28

29. © 2016 Continuum Analytics - Confidential & Proprietary 29 Data ScientistBiz Analyst Data EngineerDeveloper DevOps Modern Data Science Teams
 Love ANACONDA • Hadoop / Spark • Programming Languages • Analytic Libraries • IDE • Notebooks • Visualization • Spreadsheets • Visualization • Notebooks • Analytic Development Environment • Database / Data Warehouse • ETL • Programming Languages • Analytic Libraries • IDE • Notebooks • Visualization • Database / Data Warehouse • Middleware • Programming Languages

30. © 2016 Continuum Analytics - Confidential & Proprietary 30 Anaconda Powers Teams EXPLORE 
 & ANALYZE COLLABORATE 
 & PUBLISH DEPLOY &
 OPERATE • Explore & prepare data • Build, test, validate data science models with Python & R • Build simulations & optimizations • View data lineage & reuse transformations • Leverage & explore metadata • Create & share data science notebooks with interactive visualizations • Identify reusable data science assets easily • Authorize access to data science projects • Manage & control data science asset versions • Build & share data science packages & environments • Launch & provision distributed environments

31. © 2016 Continuum Analytics - Confidential & Proprietary 31 Write Once, Deploy AnywhereOPENDATASCIENCE Explore & Analyze Collaborate & Publish Deploy & Operate Servers Linux, Windows OSX GPUs & High End Workstations Linux & Windows NVIDIA, AMD, X86/ARM Clusters Yarn, Mesos, MPI Power8, LSF, Sun Grid Engine NoSQL MongoDB Cassandra / DataStax Hadoop Cloudera, Hortonworks Apache Hadoop & Spark Files Microsoft Excel Trifacta, Import.io DW & SQL Any SQL DB Any SQL DW, Impala

32. © 2016 Continuum Analytics - Confidential & Proprietary 32 Anaconda Architectures ON-PREMISE PRIVATE CLOUD ANACONDA CLOUD

33. © 2016 Continuum Analytics - Confidential & Proprietary 33 Public Anaconda Repository Cloud Has access to Gateway Repo Have access to Prod Repo Active Directory/ LDAP Optional Authentication Mirror Anaconda Repository Multi-Step Process – Mirror packages from Anaconda’s public Repository to a ‘Gateway’ Repo – Testers (with authorization to access Gateway) evaluate new packages. – Approved packages are mirrored to the Production Repo Server – Standard End users now have access to updated, approved packages. Gateway 
 (Test) Repo Server Production Repo ServerMirror If an Anaconda repo can function as a gateway “Tester” End User </> End User

34. © 2016 Continuum Analytics - Confidential & Proprietary 34 Public Anaconda Repository Cloud conda install numpy ipython conda update ipython conda create –n env1 ipython pandas conda env upload environment.yml project1 anaconda notebook upload project1.ipynb conda build project2 anaconda upload project2.bz2 Active Directory/ LDAP Optional Authentication Firewall Anaconda Repository—Air-Gapped Install On-site Package Repo and Sharing platform – Mirror public repository of packages – Analysts consume packages from local repo – Analysts upload and share notebooks & pre-configured computing environments – Developers create, deploy & share custom packages Internal Anaconda Repository
 (pre-loaded from disk) Analyst 1 Analyst 2 </> Developer

35. © 2016 Continuum Analytics - Confidential & Proprietary 35 Internal Anaconda Repository Package Control Head Node Cluster Provisioning Job Submission Worker
 Nodes Edge Node State Management Job Control Package Control Cluster Anaconda Scale: Cluster Management Client Machine

36. © 2016 Continuum Analytics - Confidential & Proprietary 36 Gateway & Project Nodes,
 running IPython kernels Package Control Internal Anaconda Repository Authentication Anaconda Enterprise Notebook Server Computation Web Interface Active Directory/ LDAP Optional Workflow: – Analyst Log into the Enterprise notebook server, authenticating against LDAP/AD – Based on the project they select, is re-directed to the appropriate project node – All notebooks/python code runs on project nodes; any needed packages are pulled down from your local repository Anaconda Enterprise Notebook Computing User 1 User 2 User 3

37. © 2016 Continuum Analytics - Confidential & Proprietary 37 Client Machine Internal Anaconda Repository Package Control Head Node Cluster Provisioning Job Submission Hadoop Worker Nodes Edge Node State Management Cluster Package Control Authentication Anaconda Enterprise Notebook Server Web Interface Computation Package Control LDAP: TCP 389/636 HTTP: TCP 8080 HTTP: TCP 5002 SSH: TCP 22 SALT: TCP 4505, 4506 HTTP/HTTPS: TCP 80/44 TCP 8080 Teradata Integrated Environment User 1 User 2 User 3Analyst 1 Analyst 2 Developer </>

38. © 2016 Continuum Analytics - Confidential & Proprietary 38 Anaconda
 Accelerates Adoption of Open Data Science for Enterprises Across all Data, Operating Systems, & Hardware Platforms Explore & Visualize complex data easily Harness Open Source Python & R Analytics Write Once, Deploy Anywhere for Scalable High Performance Data Engineering Simplified for All Data Collaborate with Your Team anywhere in the World Integrate Data from Anywhere

Add a comment