Turbo charge your logs

67 %
33 %
Information about Turbo charge your logs

Published on February 27, 2014

Author: jeremycook0

Source: slideshare.net


Talk on intelligently consuming log messages given at ConFoo on Feb 27th 2014

Turbo charge your logs

Who? ● ● ● ● ● Ex-pat Englishman, now living in Southern Ontario. Web developer for 5 years, mostly PHP. (Almost) senior software engineer at TribeHR. Co-organiser of Guelph PHP User Group. Ex-professional musician.

Why logging?

Your app is trying to talk to  you! ● ● ● ● Logs are the way your app speaks to you. Ignore log messages at your peril... Logging is the L in LUCID development. http://crisscott.com/2012/09/11/lucid-developme

It's confessional time...

Using log data is hard ● Not in a human friendly format. ● A lot of data. ● Many log files. ● Potentially many servers.

Use your logs pro­actively How can we stop using log data reactively and start using it pro-actively?

The 'ideal' logging setup ● ● Centralised. Accepts logs from application code, software and the OS. ● Performant. ● Scalable. ● Easily searchable. ● Alarms and alerting.

RFC 5424 logging levels ● Debug ● Info ● Notice ● Warning ● Error ● Critical ● Alert ● Emergency

Logging from your code

What's wrong with  error_log? ● Nothing at all but... ● It's limited: – Have to format the message yourself. – Limited number of destinations. – Doesn't support logging levels defined in RFC 5424.

Introducing Monolog ● PHP 5.3+ logging library by Jordi Boggiano based on Python's Log Book library. ● PSR-3 compliant. ● Supports RFC 5424.

Installing Monolog ● ● ● Symfony2, Laravel4, Silex and PPI all come with Monolog. CakePHP and Slim have have plugins to use it. Most easily installed with Composer:

Monolog concepts ● Channels. ● Handlers. ● Formatters. ● Processors.

Channels ● ● ● A channel is a name or category for a logger. Each logger instance is given a channel when instantiated. Allows for multiple loggers, each with a different channel.

Handlers ● ● ● ● Handlers write log messages to a storage medium. Multiple handlers can be attached to each logger. Set lowest level handler logs at and if it 'bubbles'. Many handlers available or you can write your own.

Example handlers Files/Syslog Notifications ● Stream Handler ● Mail handlers ● Rotating File Handler ● Pushover Handler ● Syslog Handler ● HipChat Handler Debugging Networked Logging ● Socket Handler ● AMQP Handler ● Gelf Handler ● Zend Monitor Handler ● ● FirePHP Handler ChromePHP Handler

Formatters ● ● Processes a log message into the appropriate format for a handler. Each handler has a default formatter to use but this can be overridden.

Simple example

Using multiple handlers

Leveraging bubbling

Processors ● ● ● Used to amend or add to the log message. PHP callable, called when a message is logged. Built in processors available: – IntrospectionProcessor – WebProcessor – MemoryUsageProcessor – MemoryPeakUsageProcessor – ProcessIdProcessor – UidProcessor

Processor example

Where does this get us to? ● ● Centralised. Maybe... Accepts messages from application code, software and the OS. ● Performant. Maybe... ● Scalable. Maybe... ● Easily searchable. ● Alarms and alerting. Yes but crude.

We can do better!

Leveraging Syslog

Why Syslog? ● ● Loggable events don't only happen in code! To get a full picture of what's going on we need to monitor what's going on in other services too.

Syslog basics ● ● ● ● OS daemon to process log messages. Messages are assigned a facility, such as auth, authpriv, daemon or cron or a custom one. Messages are also assigned a severity, defined in RFC 5424. Messages can be sent to files, console or a remote location.

Which Syslog daemon  to use? ● In part will depend on your OS. ● Things to consider: – Syslog is the oldest with not as many features. – Syslog-ng is produced under a dual license. – Rsyslog fully featured and open source.

Introduction to Rsyslog ● Fork of syslog by Rainer Gerhards. ● Drop in replacement for syslog. ● ● Many, many features including plugin system for extending. Default syslogger in Debian, can be installed on other distros too.

Remote logging with  Rsyslog ● Rsyslog can be configured to work in a client-server setup. – – ● One or more machines are setup as clients to forward log messages. One machine is setup to receive and store them. Probably want to filter sender on the receiving machine...

Rsyslog client setup

Rsyslog server setup

Leveling up with Rsyslog ● ● Apache can send all error logs to syslog directly. Rsyslog can also monitor other log files using the Text File Input module. – Example of monitoring Apache access log at https://gist.github.com/joseph12631/2580615

Where does this get us? ● ● Centralised. Yes. Accepts messages from application code, software and the OS. Possibly. ● Performant. Depends. ● Scalable. Depends. ● Easily searchable. ● Alarms and alerting. Yes but crude.

Taking it further with  Logstash

What is Logstash? ● ● ● Tool to collect, filter and output log messages. Built in web interface or richer web interface project called Kibana available. Full information at http://logstash.net/ and Kibana demo at http://demo.logstash.net/

Installing Logstash ● ● Current release is 1.3.3 and can be downloaded from here. Run from cli, use supervisord or an init.d/upstart script (cookbook entry on how to do this at http://cookbook.logstash.net/).

Inputs, filters and outputs Inputs – AMQP/RabbitMQ – Syslog – Varnishlog Outputs – Statsd – XMPP – AMQP/RabbitMQ – Nagios – Graphite Filters – Anonymize – Grok – Geoip – Mutate

Logstash config ● ● ● When starting specify the path to a config file for Logstash to use. Three main sections: input, filter and output. Each section may have multiple instances of each type.

Sample configuration file input { file { path => "/var/log/apache2/*access.log" type => "apache" } } filter { if [type] == "apache" { grok { pattern => "%{COMBINEDAPACHELOG}" } } } output { redis { host => "" data_type => "list" key => "logstash" } } ● See http://michael.bouvy.net/blog/en/2013/11/19/collect-visualize-your-logs-logstash-elasticsearch-redis-kibana/

Where does this get us? ● ● Centralised. Yes. Accepts messages from application code, software and the OS. Yes. ● Performant. Yes. ● Scalable. Yes. ● Easily searchable. Possibly. ● Alarms and alerting. Yes.

Introducing Graylog2

What is Graylog2? ● ● ● ● Log storage and search application. Can accept thousands of messages per second and store terabytes of data. Web interface for searching and analytics. Built in alerting and metrics.

Installing Graylog2 ● Components: – – Graylog2 server – ● MongoDb – ● Elasticsearch Graylog2 web interface Full info on installing at http://support.torch.sh/help/kb Live demo at http://public-graylog2.taulia.com/login

Getting log messages into  Graylog2 ● Can accept log messages in 3 ways: – – Syslog via UDP or TCP. – ● Graylog Extended Log Format (GELF) via UDP . AMQP. Multiple Graylog2 server instances can be run in parallel.

Graylog2 web interface ● ● Main view shows recent log messages and graphs of recent message numbers. Single message can be clicked on to view all details for it. ● Dashboard views. ● Full search functionality. ● Analytics dashboard and metrics.

Web interface view

Details of an individual  message

Dashboard view

Searches and streams ● ● ● Web interface allows fine grained searching by different fields. Frequently used searches can be saved as streams. Streams can be marked as favourites by users and can be viewed as dashboards.

Stream alarms ● ● Alarms can be sent for a stream with user defined sensitivity. Plugins for sending alarms include: – – PagerDuty – HipChat – Twilio SMS – ● Email Jabber/XMPP You can also write your own

Where does this get to? ● ● Centralised. Yes. Accepts messages from application code, software and the OS. Yes. ● Performant. Yes. ● Scalable. Yes. ● Easily searchable. Yes. ● Alarms and alerting. Yes.

Thanks for listening ● Contact me: – – ● jeremycook0@gmail.com @JCook21 Questions?

Putting it all together A few possible implementations.

Add a comment

Related presentations

Related pages

Turbo Charge Your Practice - Log in

Email Password. or forgot my password; new: register here
Read more

Turbo charge your logs (2014) | ConFoo.ca

Many of us only look at our log files during an outage or crisis. Wouldn't it be great if they could be mined in real time? This talk will show you how to ...
Read more

Forgot Password - Turbo Charge Your Practice - Log in

Forgot Password My Email Address. If you still have problems logging in, please let us know.
Read more

Abstract for a talk entitled 'Turbo charge your logs ...

The log files our applications produce contain a wealth of information about how things are performing at any moment, yet for many of us the only time we ...
Read more

How To Turbo Your Car [In 5 Minutes] - YouTube

In this video, the boys from Mighty Car Mods show you in 5 minutes whats involved in turbo charging your car. For more information about each ...
Read more

ranchpower.com :: Ranch Power - Turbo Charge Your Diesel ...

Title: Ranch Power - Turbo Charge Your Diesel Engine. Description: ranchpower.com gets 1.81 page visits from SEs monthly through organic keywords ...
Read more

Turbocharge - definition of turbocharge by The Free Dictionary

Define turbocharge. turbocharge synonyms, ... turbo- + charge] ... Feed your brain, feed a hungry child.
Read more

Turbocharge your string keyed hashmaps - /var/log/mind

Thu 17 April 2008 | tags: java ruby performance. This post gives you a small tip which just might make a world of difference to your java hashmap's ...
Read more