Kubernetes and Prometheus

43 %
57 %
Information about Kubernetes and Prometheus

Published on July 9, 2016

Author: weaveworks

Source: slideshare.net

1. Prometheus & Kubernetes Lessons Learnt Tom Wilkie, Weaveworks 8th July 2016

2. Deploying on Kubernetes Service-oriented monitoring Alerting on differences

3. Deploying on Kubernetes

4. Not a new topic! “Monitoring Kubernetes with Prometheus” - Brian Brazil “Prometheus and Kubernetes up and running” - Fabian Reinartz Even example config upstream

5. Kubernetes concepts Pod Container Container Container Pod Container Container Container Pod Container Container Container Replication Controller / Deployment Service

6. Monitoring Pod Promtheus Container AlertManager Container Grafana Container

7. Monitoring Pod Prometheus Container AlertManager Container Grafana Pod Grafana Container Prometheus ConfigMap Grafana ConfigMap

8. Grafana Pod Grafana Container Grafana ConfigMap

9. Grafana Pod Grafana Container SOCKS Proxy SOCKS Proxy

10. Grafana Pod Grafana Container AuthFE AuthFE Container

11. Demo

12. Deploying on Kubernetes

13. Service-oriented monitoring

14. When configuring Prometheus: • map Kubernetes Services to Prometheus Jobs • map Kubernetes Pods to Prometheus Instances Using relabelling: relabel_configs: - source_labels: [__meta_kubernetes_service_namespace] action: replace regex: ^(.+)$ target_label: namespace replacement: $1 - source_labels: [__meta_kubernetes_service_name] regex: ^(.+)$ target_label: job replacement: $1

15. If you get this - source_labels: [__meta_kubernetes_role] action: keep regex: endpoint

16. But not all my services export metrics! - source_labels:[__meta_kubernetes_service_annotation_prometheus_io_scrape] action: drop regex: false And add this to your service definition: apiVersion: v1 kind: Service metadata: annotations: prometheus.io.scrape: "false"

17. What about exporters? Memcached Pod memcached Container memcached-exporter Container

18. What about exporters? - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_i action: replace target_label: __address__ regex: ^(.+)(?::d+);(d+)$ replacement: $1:$2 And add this to your service definition: apiVersion: v1 kind: Service metadata: annotations: prometheus.io.port: "9150"

19. Service-oriented monitoring

20. Alerting on differences

21. What I really came here to talk about…

22. Kubernetes best-practice is to version control your config. Whats running vs whats checked in with drift unless we monitor it.

23. $ ./kubediff Usage: kubediff [options] <dir/file>... Compare yaml files in <dir> to running state in kubernetes and print the differences. This is useful to ensure you have applied all your changes to the appropriate environment. https://github.com/weaveworks/kubediff

24. Run it as a service: This will export the exit code as a Prometheus metric

25. Get alerts from it: https://github.com/weaveworks/kubediff ALERT Kubediff IF max(command_exit_code{job="kubediff"}) != 0 FOR 2h

26. Alerting on differences

27. Deploying on Kubernetes Service-oriented monitoring Alerting on differences

28. Thanks! Questions? @tom_wilkie

Add a comment

Related pages

Prometheus and Kubernetes up and running - CoreOS

After a few seconds, the Prometheus pod should be up and running. We can use any node in our cluster to access Prometheus on service port 30900.
Read more

Hands on: Monitoring Kubernetes with Prometheus - CoreOS

Hands on: Monitoring Kubernetes with Prometheus ... This is more convenient to type and more flexible across changes. Note that the equals sign after image ...
Read more

Kubernetes-Monitoring: Prometheus in Version 1.0 ...

Die APIs des Monitoring-Systems der Cloud Native Computing Foundation gelten nun als stabil, was auch der Sprung auf die Version 1.0 implizieren soll.
Read more

Configuration | Prometheus

Prometheus is configured via command-line flags and a configuration file. While the command-line flags configure immutable system parameters (such as ...
Read more

Prometheus 1.0: "Prometheus and Kubernetes share spiritual ...

In this interview, Björn Rabenstein, engineer at SoundCloud and Prometheus core developer, talks about the features benefits of Prometheus 1.0.
Read more

Prometheus: Erste Vollversion des Monitoring-Tools - JAXenter

Wer Kubernetes in Verbindung mit Prometheus nutzen möchte, hat Glück: Das Orchestrierungs-Tool von Google unterstützt Prometheus bereits nativ, sodass ...
Read more

Google And Friends Add Prometheus To Kubernetes Platform

Google And Friends Add Prometheus To Kubernetes Platform. May 9, 2016 Timothy Prickett Morgan. There are a lot of moving parts in a modern platform, and in ...
Read more

prometheus/prometheus-kubernetes.yml at master ...

# A scrape configuration for running Prometheus on a Kubernetes cluster. # This uses separate scrape configs for cluster components (i.e. API server, node)
Read more

Prometheus and Kubernetes up and running | Hacker News

Prometheus and Kubernetes up and running (coreos.com) ... For the most part we are recommending folks use Kubernetes for new projects over fleet.
Read more

Prometheus: Service Discovery with Kubernetes - Movio Blog

The third in our blog series, Prometheus: Service Discovery with Kubernetes.
Read more