MQTT. Kafka. InfluxDB. SQL. IoT Harmony. #tutorial by Stefan Bocutiu

50 %
50 %
Information about MQTT. Kafka. InfluxDB. SQL. IoT Harmony. #tutorial by Stefan Bocutiu

Published on February 27, 2018

Author: landoop

Source: slideshare.net

1. Stefan Bocutiu @stepib London Apache Kafka Meetup, Feb 2018 End to end topologies on Apache Kafka 
 An IoT use case CONNECT ANALYSE REACT

2. About Today Intros IoT High Level Architecture Build the Pipeline Q&A

3. Hi! It’s Stefan!
 @stepib
 https://github.com/stheppi CONNECT ANALYSE REACT Streaming Data Processing & Integration
 KAFKA - KUBERNETES

4. EVOLVESCALEBUILDIoT initiative challenges ★ 94% face challenges collecting and analysing IoT data ★ 26+ Billion devices by 2020, Gartner More Data, More Complexity ★ 2.500.000 trillion bytes produced daily REAL TIME DATA INGESTION + IMMEDIATE QUERIES = 
 ACTIONABLE / TIMELY INSIGHTS IoT Data Flows

5. IoT data flow store datastream & process datasensor data

6. IoT data flow store datastream & process datasensor data E T L

7. InfluxDBsensor data MQTT InfluxDB 
 Connector LENSES SQL PROCESSOR KAFKA 
 CONNECT KAFKA 
 CONNECT MQTT 
 Connector KAFKA STREAMS

8. InfluxDBsensor data MQTT InfluxDB 
 Connector LENSES SQL PROCESSOR KAFKA 
 CONNECT KAFKA 
 CONNECT MQTT 
 Connector KAFKA STREAMS Synthetic Data With Moquette
 1 Broker, 
 data generate sensor data InfluxDB Docker

9. ® IoT & High Volumes, N devices
 & irregular intervals Real Time Analytics 
 & Microservices Unlimited streams of data, async transfers
 Producers & Consumers decoupled operation Kafka Streams API
 Processing and analysing data in motion Multiple sources of data 
 & long term storeage Kafka Connect API, 
 move data with pluggable reusable & scalable connectors An open source streaming framework with messaging semantics where records are key-value pairs.

10. LENSES BOX
 For Kafka Development Single docker image with all services Growing list of examples & synthetic data generators. Stream Reactor 25+ connectors Kafka CLI autocomplete. Rest and WebSocket endpoints Redux JS Client Testing Libraries FREE! All you need to start working with Apache Kafka ecosystem

11. MQTT Source connector
 • Message Queueing Telemetry Transport messaging protocol • Optimised for connected devices and M2M protocol • Low bandwidth, high-latency, unreliable networks • Plain or SSL connections • Kafka Connect Query Language INSERT INTO kafka_topic SELECT * FROM /mqtt_source_topic [WITHCONVERTER=`myclass`] MQTT Distributed, Fault tolerant, scalable No code required CLI / UI / Monitor KCQL Support Multiple Distributions

12. • Process data in motion with Lenses SQL • Create & Deploy Lenses SQL Processors • Native Kafka & Kubernetes Support Scalability • Visual Topology • Monitor and manage processors via LENSES Process & Analyse Data in motion

13. InfluxDB Sink connector
 • AVRO & JSON Support • Error handling ( Noop, Retry, Throw) • Field selection & field alias • Multiple inserts supported • Kafka Connect Query Language • MEASUREMENTS, KCQL: INSERTS • TAGS, KCQL: WITHTAG • TIMESTAMPS, KCQL: WITHTIMESTAMP • DURATION Distributed, Fault tolerant, scalable No code required CLI / UI / Monitor KCQL Support Multiple Distributions INSERT INTO sensorMeasure 
 SELECT * FROM sensor_data_avg 
 WITHTAG (id, poweredBy=Lenses)

14. Distributed, Fault tolerant, scalable No code required CLI / UI / Monitor KCQL Support Multiple Distributions InfluxDB Sink connector
 • Timestamp nanosecond • KCQL • No filter support • No function support • Throughput involves domain knowlege • Kafka Connect rebalances

15. visual data flow Visual and Interactive Topology
 for a 360 view of your data End-To-End Topology Management
 Visualise your Kafka-based data flows
 in real time

16. Wrap it up! CONNECT ANALYSE REACT Real Time, IoT pipelines in minutes! MQTT

17. Contact us www.landoop.com info@landoop.com Twitter @LandoopLtd Github github.com/landoop Gitter https://gitter.im/Landoop Slack https://launchpass.com/landoop-community Thank you!

Add a comment