advertisement

Big data infrastructure todo-tasks Rfx Framework

50 %
50 %
advertisement
Information about Big data infrastructure todo-tasks Rfx Framework
Technology

Published on March 12, 2014

Author: tantrieuf31

Source: slideshare.net

Description

Big data infrastructure todo-tasks Rfx Framework
advertisement

Overview of Rfx Framework / Platform https://docs.google.com/document/d/1wutns90tuW1PGR03tXhDE_­DkrdWZtfvh9R_cJRtrXk/edit?usp=sharing Big Data Infrastructure - TODO Tasks Update March 12, 2014 by Triều (@tantrieuf31) ● Module HTTP Log Server: ○ Hot deployment/restart/shutdown Http Log Server ○ Reactive streaming for Kafka Producer (RxJava)  ■ https://github.com/Netflix/RxJava/wiki/Transforming­Observables ● Module Messaging (Kafka): https://bitbucket.org/trieunt/kafka ○ Tìm 1 cơ chế quản lý configs và rotate kafka logs 1 cách an toàn hơn (hiện đang bị 1 issue  Kafka Consumer chưa đọc xong mà Kafka log đã move đi => kg tìm thấy offset để đọc tiếp =>  thiếu data) ○ Dự đoán tốc độ tăng file Kafka log để chọn 1 configs tối ưu cho từng loại sản phẩm  (machine learning (linear regression) for system performance) ○ Tạo mapping (thời gian, offset và binary offset files) (lúc cần parse lại thì dễ tìm files) ○ Quản lý + index lại offset của Kafka theo thời gian (giờ, ngày, ...), lúc cần thì set vào là chạy  reparse lại (hiện chưa implement) ● Module Stream Data Processing: https://bitbucket.org/trieunt/rfx/wiki/Home ○ Quản lý memory của worker node (nếu set HeapSize quá thấp => Worker sẽ die/restart liên  tục do kg đủ memory để chạy vì log nhiều) ○ Cơ chế extensions/plugins/hooking  vào hệ thống (phân chia core và applications) ○ Refactoring (tổ chức lại code cho rõ ràng) giữa logic code công việc giữa:  ■ parse => ghi vào Redis (chỉ parse, counting và check rules) ■ parse => ghi ra raw log files trong 1 worker (chỉ parse và write raw logs) ○ Unit Test Tools (Kafka Producer) + Test Tools (integration test) cho Reactive Topologies  ○ Cải thiện chức năng debug log của Worker (ElasticSearch+Kibana) ○ Monitor Front End cho tất cả các critical metrics: ■ worker nodes (logs, memory, restart time, running, died, uptime, downtime ) ■ alert/notification ■ số lượng log đọc từ Kafka, parsed OK, check OK, save OK ■ Disk Free, memory cho worker ■ Backup Redis Data ■ Simple Analytics Dashboard cho logs (analytics) ○ New Job Server (dùng Groovy script để dễ deploy và control qua Pub/Sub Redis) ■ Synchronized Data job ● Module Active Intelligence (tính năng mới ) ● social data crawler Facebook/Twitter/Google+ (Rfx Social Data Crawler) ● Clustering Stream Data (test case: tin tức về các vụ tai nạn xe cột / cướp giật / thảm họa thiên  nhiên) ­ dùng Apache Spark http://spark.apache.org ● Realtime Visualization Engine with HTML5 Web Socket (D3.js + Netty + Akka Actor)

Add a comment

Related presentations

Presentación que realice en el Evento Nacional de Gobierno Abierto, realizado los ...

In this presentation we will describe our experience developing with a highly dyna...

Presentation to the LITA Forum 7th November 2014 Albuquerque, NM

Un recorrido por los cambios que nos generará el wearabletech en el futuro

Um paralelo entre as novidades & mercado em Wearable Computing e Tecnologias Assis...

Microsoft finally joins the smartwatch and fitness tracker game by introducing the...

Related pages

Building infrastructure for Big Data - Technology

Big Data for Big Power: How smart is the grid if the infrastructure is stupid?
Read more

RFX 8220 - Documents

Instruction manual for Rockford Fosgate RFX 8210, 8220, ... Share RFX 8220. ... Big data infrastructure todo-tasks Rfx Framework.
Read more

Technology - A Business Data Intelligence Company

BDI's Framework; Tower Infrastructure ... Sizing; Tableau Services; IOT services; Big Data. ... Blogs. Company: About Us: Career in BDI: Technology:
Read more

About BDI Systems - A Business Data Intelligence Company

... Big Data Engineering, Business Intelligence, ... of Big Data and Hadoop based infrastructure, ... a BI Visualization framework where charts and ...
Read more

Big Data/Hadoop Infrastructure Considerations - Documents

Planning big-data infrastructure, implications for. Docslide.us. ... Big Data in a Software-Defined DatacenterRichard McDougallChief Architect, ...
Read more

Advisory Platform Architect Description at EMC CORPORATION

Advisory Platform Architect Job Location: Germany ... leveraging big and fast data, ... Participate in RFx and POCs.
Read more

Freiberufler: Data Warehouse, Business Intelligence, BI ...

Bewerber für Festanstellungen (ehemals Randstad Professionals) Bitte hier anmelden. Noch nicht Mitglied? Jetzt registrieren. Experten finden
Read more