Next-Generation BPM - How to create intelligent Business Processes thanks to Big Data and Apache Hadoop

67 %
33 %
Information about Next-Generation BPM - How to create intelligent Business Processes...

Published on February 6, 2014

Author: KaiWaehner

Source: slideshare.net

Next-Generation BPM – How to create intelligent Business Processes thanks to Big Data Talend, Global Leader in Open Source Integration Solutions Kai Wähner kontakt@kai-waehner.de @KaiWaehner Xing / LinkedIn www.kai-waehner.de

Kai Wähner Main Tasks Requirements Engineering Enterprise Architecture Management Business Process Management Architecture and Development of Applications Service-oriented Architecture Integration of Legacy Applications Cloud Computing Big Data Contact Consulting Developing Coaching Speaking Writing © Talend 2013 Email: kontakt@kai-waehner.de Blog: www.kai-waehner.de/blog Twitter: @KaiWaehner Social Networks: Xing, LinkedIn “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner

Key messages BPM should be used (just) for optimizing business processes! Intelligent business processes need big data and integration! Big data will reduce human interactions in BPM further! © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner

Agenda • Big data paradigm shift • Use cases for big data • Intelligent business processes • Technology and product perspective • Implementation of an use case © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner

Agenda • Big data paradigm shift • Use cases for big data • Intelligent business processes • Technology and product perspective • Implementation of an use case © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner

Why should you care about big data? “If you can't measure it, you can't manage it.” William Edwards Deming (1900 –1993) American statistician, professor, author, lecturer and consultant © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner

Why should you care about big data? „Silence the HiPPOs“ (highest-paid person‘s opinion)  Being able to interpret unimaginable large data stream, the gut feeling is no longer justified!  © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner

Why does big data exist? Changing Scale Sensors Changing Expectations Cloud Changing Interactions © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner

Three shifts in the way we analyze information • Messiness: Using ALL data, not just samples • Also bad data (e.g. Word spell checker, Google autocomplete and „did you mean...“ recommendation • Correlations: Instead of causalities • May not tell us WHY something is happening, but THAT it is happening • In many situations, this is good enough • What drug substance cures cancer? When should I buy an airplane ticket? • Datafication: Store, process, combine, reuse, enhance all data! • Digitalisation (Amazon Kindle  Read) vs. Datafication (Google Books  Read, Search, Process, ...) • Words becomes data: Google books: not just read, but also search, analyse, etc. • Locations becomes data: GPS: not just navigation, but also insurance costs, economic routes, etc. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner

The Vs of big data Volume (terabytes, petabytes) Velocity (realtime or nearrealtime) Variety (social networks, blog posts, logs, sensors, etc.) © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Value

Big data tasks to solve - before analysis Big Data Integration – Land data in a Big Data cluster – Implement or generate parallel processes Big Data Manipulation – Simplify manipulation, such as sort and filter – Computational expensive functions Big Data Quality & Governance – Identify linkages and duplicates, validate big data – Match component, execute basic quality features Big Data Project Management – Place frameworks around big data projects – Common Repository, scheduling, monitoring © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner

Agenda • Big data paradigm shift • Use cases for big data • Intelligent business processes • Technology and product perspective • Implementation of an use case © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner

Storage  Reduce costs Global Parcel Service A lot of data must be stored „forever“ ➜ Numbers increase exponentially ➜ Goal: As cheap as possible ➜ Problem: (Fast) queries must still be possible ➜ Solution: Commodity servers and „Hadoop querying“ ➜ http://archive.org/stream/BigDataImPraxiseinsatz-SzenarienBeispieleEffekte/Big_Data_BITKOM-Leitfaden_Sept.2012#page/n0/mode/2up © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner

Replace ETL Improve performance “The advantage of their new system is that they can now look at their data [from their log processing system] in anyway they want: ➜ Nightly MapReduce jobs collect statistics about their mail system such as spam counts by domain, bytes transferred and number of logins. ➜ When they wanted to find out which part of the world their customers logged in from, a quick [ad hoc] MapReduce job was created and they had the answer within a few hours. Not really possible in your typical ETL system.” http://highscalability.com/how-rackspace-now-uses-mapreduce-and-hadoop-query-terabytes-data © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner

Risk management  Customer success Deduce Customer Defections http://hkotadia.com/archives/5021 © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner

Flexible pricing  Increase revenue ➜ ➜ ➜ ➜ With revenue of almost USD 30 billion and a network of 800 locations, Macy's is considered the largest store operator in the USA Daily price check analysis of its 10,000 articles in less than two hours Whenever a neighboring competitor anywhere between New York and Los Angeles goes for aggressive price reductions, Macy's follows its example If there is no market competitor, the prices remain unchanged http://www.t-systems.com/about-t-systems/examples-of-successes-companies-analyze-big-data-in-record-time-l-t-systems/1029702 © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner

Great big data use cases, but ... ➜ ➜ ➜ ➜ How do you put this big data easily in the hands of the people that need it? Making the data “actionable” is the real challenge. Seeing the information that helps make a decision on a composite dashboard is just the first step and where too many companies stop. A business must be able to fire off the business process to execute the decision made regarding the data. http://smartdatacollective.com/matt-davies/104576/data-driven-bpm-making-big-data-actionable © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Kai Wähner

Agenda • Big data paradigm shift • Use cases for big data • Intelligent business processes • Technology and product perspective • Implementation of an use case © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner

Intelligent business processes Humans have to interpret large data to make decision. Using gut feeling is nothing but gambling. ➜ Just doing big data analytics is not enough. Systematic and monitored human interactions are as important to get best outcomes. ➜ An intelligent business process combines big data and BPM. This enables humans to make data-driven decisions based on big data analytics. ➜ © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner

Intelligent business processes ➜ Process starts action (PULL Big Data) • Manual or automated • Faster responses (e.g. „spam by domain“) • Better outcomes (e.g. „recommendation engine“) ➜ Data starts action (Big Data PUSH) • (Usually) automated • Predictive processes (e.g. „preventing flu epidemic“) • Handle before it happens (e.g. „customer deduction“) © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner

Combination of big data and BPM How are they related? ➜ How to combine? ➜ How to realize this technically? ➜ © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner

How BPM? Script Task Groovy JavaScript etc. Service Task SOAP Web Service Everything from Cobol to Ruby... © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Kai Wähner ... or a „big data service“

Challenge Separation of Concerns © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Kai Wähner

Building Blocks for „Intelligent Business Processes“ Integration • ETL • Connectivity / adaptors to connect to external systems using a variety of different protocols • Predefined EIP for message routing Big Data • Processing • Analytics BPM • Do queries to make decisions • Human or machine © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Kai Wähner

Agenda • Big data paradigm shift • Use cases for big data • Intelligent business processes • Technology and product perspective • Implementation of an use case © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner

Building Blocks for „Intelligent Business Processes“ Integration • Extract Transform Load (ETL) • Connectivity / adaptors to connect to external systems using a variety of different protocols • Predefined EIP for message routing Big Data • Processing • Analytics BPM • Do queries to make decisions • Human or machine © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Kai Wähner

Enterprise Integration Patterns © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner

Do not write all that “glue code”! AmazonS3 s3 = new AmazonS3Client(new PropertiesCredentials( S3Sample.class.getResourceAsStream("AwsCredentials.properties"))); String bucketName = "my-first-s3-bucket-" + UUID.randomUUID(); String key = "MyObjectKey"; try { s3.createBucket(bucketName); s3.putObject(new PutObjectRequest(bucketName, key, createSampleFile())); S3Object object = s3.getObject(new GetObjectRequest(bucketName, key)); ObjectListing objectListing = s3.listObjects(new ListObjectsRequest() .withBucketName(bucketName) .withPrefix("My")); s3.deleteObject(bucketName, key); s3.deleteBucket(bucketName); } catch (AmazonServiceException ase) { // error handling... } catch (AmazonClientException ace) { // error handling... } © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner

Integration framework (e.g. Apache Camel) // Producer from(“ftp:toS3") .setHeader(S3Constants.KEY, simple(“order.txt")) .to("aws-s3://myBucket?accessKey=" + a+ "&secretKey= " + s) // Consumer from(„salesforce://orders__c?user=dummy1“) .filter(„customer == ${dummyCustomer}) .to(“ibm-database:orderData") © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner

Enterprise Service Bus (e.g. Talend ESB) © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner

Alternatives for integration Integration Suite Enterprise Service Bus Integration Framework Low Connectivity Routing Transformation © Talend 2013 High + INTEGRATION Tooling Monitoring Support + BUSINESS PROCESS MGT. BIG DATA / MDM REGISTRY / REPOSITORY RULES ENGINE „YOU NAME IT“ “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Complexity of Integration

Building Blocks for „Intelligent Business Processes“ Integration • ETL • Connectivity / adaptors to connect to external systems using a variety of different protocols • Predefined EIP for message routing Big Data • Processing • Analytics BPM • Do queries to make decisions • Human or machine © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Kai Wähner

Technology perspective How to process big data? © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner

How to process big data? The defacto standard for big data processing © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner

How to process big data? “A big part of [the company’s strategy] includes wiring SQL Server 2012 (formerly known by the codename “Denali”) to the Hadoop distributed computing platform, and bringing Hadoop to Windows Server and Azure” Even Microsoft (the .NET house) relies on Hadoop since 2011 © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner

What is Hadoop? Apache Hadoop, an open-source software library, is a framework that allows for the distributed processing of large data sets across clusters of commodity hardware using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner

How to process big data? © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner

Hadoop alternatives Integration Suite Hadoop Distribution Apache Hadoop few MapReduce HDFS Ecosystem © Talend 2013 many + Packaging Deployment-Tooling Support Tooling / Modeling Code Generation Scheduling Other Tools (ESB, BPM, ...) + “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Features included

(Near) Realtime? ! Hadoop cannot solve every big data problem. Complex event processing and real-time analytics have to be solved in another way (at least today). In-memory computing and streaming platforms are good alternatives or complements to Hadoop for processing and analyzing big data. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner

Building Blocks for „Intelligent Business Processes“ Integration • ETL • Connectivity / adaptors to connect to external systems using a variety of different protocols • Predefined EIP for message routing Big Data • Processing • Analytics BPM • Do queries to make decisions • Human or machine © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Kai Wähner

Standards jPDL BPEL BPM BPMN XPDL WF-XML ARIS EPC BPEL4People © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Kai Wähner

BPMN „Business Process Model and Notation (BPMN) is a graphical representation for specifying business processes in a business process model.“  BPMN 2.0 is also executable! Wikipedia © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Kai Wähner

Alternatives for BPM Integration Suite BPM Suite BPM Framework Low High Coding Service Tasks Human Interaction GUI © Talend 2013 + BPM Tooling Monitoring Support + ESB BIG DATA / MDM REGISTRY / REPOSITORY RULES ENGINE „YOU NAME IT“ “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Kai Wähner Complexity of Orchestration

Building Blocks for „Intelligent Business Processes“ Let‘s realize it !!! Integration • ETL • Connectivity / adaptors to connect to external systems using a variety of different protocols • Predefined EIP for message routing Big Data • Processing • Analytics BPM • Do queries to make decisions • Human or machine © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Kai Wähner

Frameworks vs. Tools Suite of Tools Specific Tools Frameworks Low High e.g. Camel (Integration) Hadoop (Big Data) Activiti (BPM) © Talend 2013 e.g. Mule ESB (Integration) MapR (Big Data) Camunda (BPM) e.g. Talend Unified Platform i.e ALL-IN-ONE (Integration, Big Data, BPM) “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Kai Wähner Complexity of Orchestration

Custom combination of integration, big data and BPM? • A lot of glue code • Testing • Bugfixing • No support Some other people already had the problems you would have! © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner Kai Wähner

Agenda • Big data paradigm shift • Use cases for big data • Intelligent business processes • Technology and product perspective • Implementation of an use case © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner

Flexible pricing  Increase revenue ➜ ➜ ➜ ➜ With revenue of almost USD 30 billion and a network of 800 locations, Macy's is considered the largest store operator in the USA Daily price check analysis of its 10,000 articles in less than two hours Whenever a neighboring competitor anywhere between New York and Los Angeles goes for aggressive price reductions, Macy's follows its example If there is no market competitor, the prices remain unchanged http://www.t-systems.com/about-t-systems/examples-of-successes-companies-analyze-big-data-in-record-time-l-t-systems/1029702 © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner

Implementation of an use case Suite of Tools Specific Tools Frameworks Low High e.g. Camel (Integration) Hadoop (Big Data) Activiti (BPM) © Talend 2013 e.g. Mule ESB (Integration) MapR (Big Data) Camunda (BPM) Complexity of Orchestration e.g. Talend Unified Platform i.e ALL-IN-ONE (Integration, Big Data, BPM) “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner

Talend Unified Platform Big Data Data Quality Data Integration MDM ESB BPM  Commercial license  Subscription model  Support included  Open source license  Free of charge Big Data Data Quality Data Integration MDM ESB  Optional support  Based on open source projects such as Eclipse or Apache Camel, CXF, Hadoop © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner

Example (Talend): Integration Connect to data sources from competitors, for example via REST service, Twitter API, or custom scripts. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner

Example (Talend): Big Data Processing Move data to HDFS for processing, as your classic servers and data warehouses are not able to process this semi-structured data fast enough (and cheap), probably. Manipulate the data, in other words, filter relevant information, sort it, and compare it to prices of your products. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner

Example (Talend): Business Process Start a new instance of a business process to review the result and continue with further tasks, such as calling a web service which does the price reduction in selected locations. Reviews can be done by human interaction or via automated tasks depending on the proposed price reduction. © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner

Implementation of an use case „Talend Unified Platform“ in action... © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner

Did you get the key message? © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner

Key messages BPM should be used (just) for optimizing business processes! Intelligent business processes need big data and integration! Big data will reduce human interactions in BPM further! © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner

Did you get the key message? © Talend 2013 “How to create intelligent Business Processes thanks to Big Data” by Kai Wähner

Thank you for your attention. Questions? KAI WÄHNER kontakt@kai-waehner.de www.kai-waehner.de LinkedIn / Xing @KaiWaehner

Add a comment

Related pages

Slides from OOP 2014 Online: Next-Generation BPM – How ...

... Next-Generation BPM - How to create intelligent Business Processes thanks to Big ... Business Processes thanks to Big Data and Apache Hadoop from ...
Read more

Slides online: How to create intelligent Business ...

... create intelligent Business Processes thanks to Big ... create intelligent Business Processes thanks ... thanks to Big Data (BPM, Apache Hadoop, ...
Read more

How to Create intelligent Business Processes Thanks to Big ...

... to create intelligent Business Processes thanks ... BPM conference in London about “How to create intelligent Business Processes thanks to Big Data ...
Read more

4 Ways Big Data Will Transform Business | CSC

4 Ways Big Data Will Transform Business; ... Big data + mobile means new business processes . ... Intelligence: A Match Made in Hadoop; Big Data Meets ...
Read more

Process Orchestration | SAP - SAP Software & Solutions ...

SAP NetWeaver Process Orchestration can help ... Next Generation BPM Webinar. Hear how the intelligent business ... Big Data Business Intelligence
Read more

Business Process Management (BPM) Definition

BPM (Business Process Management) ... which describes the next generation of BPM suites. ... our Intelligent Business Process Management Suite, ...
Read more

Fast Data Analytics and an Intelligent BPM Suite for ...

An intelligent business process combines Big ... Integrate casino data from ERP stored in Apache Hadoop. ... The Next-Generation BPM for a Big Data ...
Read more

The three big data pillars: Hadoop, SAP HANA and business ...

Big data: SAP HANA and Hadoop ... preferably based on the business intelligence process model. Business ... BPM defines methods, tools and processes ...
Read more