Monitoring

Overview

To scale applications running on Kubernetes and assure quality of services one needs to gain insight on how these applications behave once deployed, observe their performance on day N and detect anomalies as soon as they happen.

In this direction, Rok ships with a full-fledged monitoring stack on Kubernetes that uses Prometheus as a collector and time series database and Grafana as a visualization and analytics solution. The goal of these services is to increase system observability, understand patterns and make the debugging process easier.

Components

Rok’s monitoring stack is based on the open-source Kube Prometheus project and consists of the following components:

Component Description
Prometheus Operator Provides Kubernetes native deployment and management of Prometheus and related monitoring components
Prometheus An open-source monitoring system with a dimensional data model, flexible query language, efficient time series database and modern alerting approach.
Node Exporter Prometheus exporter for hardware and OS metrics exposed by *NIX kernels, written in Go with pluggable metric collectors.
Kube State Metrics A simple service that listens to the Kubernetes API server and generates metrics about the state of the objects.
Alert Manager Handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integrations.
Prometheus Adapter Expose custom, application-specific metrics via Kubernetes Custom Metrics API, e.g., to configure the HPA controller
Grafana An open source visualization and analytics software that allows you to query, visualize, alert on, and explore metrics stored in various databases.

Prometheus is a full-fledged, widely-adopted monitoring system and time series database built using a HTTP pull model. It includes a dimensional data model based on labels, a custom query langue (PromQL) and an alerting system (AlertManager). Prometheus is a graduated project of the Cloud Native Computing Foundation.

The Prometheus Operator for Kubernetes introduces the monitoring.coreos.com/v1 API and manages the Prometheus and ServiceMonitor custom resources. More specifically it synchronizes the configuration of the Prometheus server based on the spec of the Prometheus CR and also ensures that metrics from all targets referred to by existing ServiceMonitors are collected.

../../../_images/prometheus_operator.png

Note

The monitoring stack that runs alongside Rok does not yet include Prometheus’s Alert Manager and Adapter. The configuration of these components depends on the needs and characteristics of each installation.

For more information regarding Prometheus features and components visit the official Prometheus Overview webpage.

Packaging

Since rok-deploy, Rok’s deployment tool for Kubernetes, is leveraging Kustomize and kubectl to deploy Rok we maintain an opinionated collection of manifests tailored for observing Rok and its external services.

Assuming that you already have a clone of Arrikto’s private deployment repository you can view the kustomization tree with Rok’s monitoring manifests under rok/monitoring.

Monitoring Targets

In this section we will describe how you can set up Prometheus to monitor your physical nodes, Kubernetes and, Rok’s external services.

Physical Nodes

To monitor the physical nodes (or cloud VMs) that host Kubernetes and, in turn, applications running on it, we need a way to gather and export critical system metrics in order to expose the overall state regarding CPU usage, Memory consumption, Disk I/O, Network traffic, etc.

The standard way to achieve this in a Prometheus-based monitoring stack is to use the Prometheus Node Exporter which runs a set of collectors for both hardware and OS metrics exposed by the kernel.

In Rok’s monitoring stack we deploy Node Exporter as a DaemonSet on Kubernetes so that we can retrieve system metrics from all nodes.

To make Prometheus aware of the Node Exporter and configure it to collect node metrics periodically we use a ServiceMonitor resource for the Node Exporter service (node-exporter.monitoring):

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    k8s-app: node-exporter
  name: node-exporter
  namespace: monitoring
spec:
  endpoints:
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    interval: 15s
    port: https
    relabelings:
    - action: replace
      regex: (.*)
      replacement: $1
      sourceLabels:
      - __meta_kubernetes_pod_node_name
      targetLabel: instance
    scheme: https
    tlsConfig:
      insecureSkipVerify: true
  jobLabel: k8s-app
  selector:
    matchLabels:
      k8s-app: node-exporter

Kubernetes

To monitor Kubernetes we need a way to gather and export metrics from core Kubernetes components, such as the API server, kubelet, scheduler, controller-manager, coreDNS, etc. These components already collect and expose metrics in the Prometheus data format via a Kubernetes Service.

Note

cAdvisor is an open-source agent that is integrated into the kubelet binary that monitors resource usage and analyzes the performance of containers. It collects statistics about the CPU, memory, file, and network usage for all containers running on a given node (it does not operate at the pod level).

In addition, we need to gather and export metrics from Kubernetes resources, such as StatefulSets, DaemonSets, PersistentVolumeClaims, etc. The standard way to achieve this in a Prometheus-based monitoring stack is to deploy kube-state-metrics in the Kubernetes cluster to be monitored. Kube state metrics focuses on generating completely new metrics from Kubernetes’ object state. It holds an entire snapshot of Kubernetes state in memory and continuously generates new metrics based off of it.

To this end, to make Prometheus aware of Kubernetes metrics (both from core components and resources) we have to create a ServiceMonitor resource for each Service that exposes metrics to be collected. Namely:

$ kubectl get servicemonitor --all-namespaces
NAMESPACE    NAME
monitoring   coredns
monitoring   kube-apiserver
monitoring   kube-controller-manager
monitoring   kube-scheduler
monitoring   kube-state-metrics
monitoring   kubelet

Rok

The integration of Rok with Prometheus is a work in progress.

Rok’s External Services

Rok depends on external services to operate. Namely, it leverages

  • etcd (key-value store for data and metadata),
  • Redis (in-memory data structure store used as cache), and
  • Istio (service mesh for microservices).

To allow Prometheus to monitor these services, i.e., scrape respective targets to collect exported metrics periodically, we have to create proper RBAC resources so that Prometheus is allowed to get, list and watch Pods, Services and Endpoints in the namespace where Rok is deployed.

Etcd

To make Prometheus aware of Rok’s etcd and configure it to scrape metrics periodically we have to create a ServiceMonitor resource for Rok’s etcd service (rok-etcd.rok).

Redis

To make Prometheus aware of Rok’s Redis and configure it to scrape metrics periodically we have to create a ServiceMonitor resource for Rok’s Redis metrics service (rok-redis-metrics.rok).

Istio

The integration of Istio with Prometheus is a work in progress.

Deploy Rok’s Monitoring Stack

To deploy Rok’s monitoring stack on Kubernetes alongside your running services you need to follow the steps below, assuming you have already prepared and configured your management environment:

  1. Go inside the local clone of Arrikto’s deployments git repository:

    root@rok-tools-0:/# cd ~/ops/deployments
    
  2. Deploy the Prometheus Operator along with needed CustomResourceDefintions:

    root@rok-tools-0:~/ops/deployments# rok-deploy --apply rok/monitoring/setup
    

    Note

    The command above creates the monitoring namespace. This is where Rok’s monitoring stack deploys its core components to.

  3. Provision persistent storage for Grafana:

    root@rok-tools-0:~/ops/deployments# kubectl apply -f rok/monitoring/install/overlays/deploy/grafana-persistentvolumeclaim.yaml
    
  4. Create a ConfigMap with the Node Exporter Full Grafana dashboard definition:

    root@rok-tools-0:~/ops/deployments# kubectl create -f rok/monitoring/install/overlays/deploy/grafana-dashboardDefinition-node-exporter-full.yaml
    

    Note

    The representation of this dashboard as a ConfigMap exceeds the character limit allowed by Kubernetes as described in this GitHub issue. That is, kubectl apply fails with:

    The ConfigMap "grafana-dashboard-node-exporter-full" is invalid:
    metadata.annotations: Too long: must have at most 262144 characters
    

    We sidestep this issue by deploying it with kubectl create.

  5. Deploy the core components of Rok’s monitoring stack:

    root@rok-tools-0:~/ops/deployments# rok-deploy --apply rok/monitoring/install/overlays/deploy
    

Verify deployment

Before proceeding to the Viewing Dashboards section you need to verify that you deployed Rok’s monitoring stack correctly and that individual components are operating normally.

  1. Ensure that all Pods are ready in the monitoring namespace:

    root@rok-tools-0:~/ops/deployments# kubectl get pods -n monitoring
    NAME                                   READY   STATUS    RESTARTS   AGE
    grafana-6d7d7b78f7-6flm7               1/1     Running   0          2m17s
    kube-state-metrics-765c7c7f95-chkzn    3/3     Running   0          2m16s
    node-exporter-zng26                    2/2     Running   0          2m16s
    prometheus-k8s-0                       3/3     Running   1          2m15s
    prometheus-operator-5f75d76f9f-fmpp5   1/1     Running   0          8m24s
    
  2. Verify that the following ServiceMonitors exist in the monitoring namespace:

    root@rok-tools-0:~/ops/deployments# kubectl get servicemonitors -n monitoring
    NAME                      AGE
    coredns                   5s
    grafana                   5s
    kube-apiserver            5s
    kube-controller-manager   5s
    kube-scheduler            5s
    kube-state-metrics        5s
    kubelet                   5s
    node-exporter             5s
    prometheus                5s
    prometheus-operator       5s
    
  3. Probe Prometheus to make sure it has detected targets for Rok’s external services in order to pull metrics periodically. For this, query the Prometheus TSDB and search for discovered targets:

    $ kubectl exec -ti -n monitoring sts/prometheus-k8s -c prometheus -- wget -qO - prometheus-k8s.monitoring:9090/metrics | grep 'discovered.*rok-etcd'
    prometheus_sd_discovered_targets{config="rok/rok-etcd-metrics/0",name="scrape"} 7
    
    $ kubectl exec -ti -n monitoring sts/prometheus-k8s -c prometheus -- wget -qO - prometheus-k8s.monitoring:9090/metrics | grep 'discovered.*rok-redis'
    prometheus_sd_discovered_targets{config="rok/rok-redis-metrics/0",name="scrape"} 7
    

Viewing Dashboards

After successfully deploying Rok’s monitoring stack you can expose the Prometheus and Grafana UIs in order to easily query Prometheus’s TSDB and view curated dashboards that visualize time series.

Prometheus UI

Prometheus offers a web UI that enables you to view simple graphs, Prometheus configuration and rules, as well as the state of the monitoring endpoints.

  1. Use kubectl to forward traffic from your local machine to Prometheus running in your Kubernetes cluster:

    root@rok-tools-0:/# kubectl port-forward --address 0.0.0.0 -n monitoring svc/prometheus-k8s 8080:9090
    

    Important

    If you are running rok-tools as a Kubernetes StatefulSet you will need a double kubectl port-forward to also forward traffic from your local machine to the rok-tools Kubernetes Service:

    $ kubectl port-forward svc/rok-tools 9090:8080
    
  2. View the Prometheus UI at localhost:9090 with your browser.

    ../../../_images/prometheus_ui.png

Grafana UI

Grafana offers a rich web UI that allows you to view or create dashboards with metrics retrieved from a multitude of datasources, set up alert rules, etc. The monitoring stack that accompanies Rok ships with certain dashboards that are pre-loaded and available upon deployment.

  1. Use kubectl to forward traffic from your local machine to Grafana running in your Kubernetes cluster:

    root@rok-tools-0:/# kubectl port-forward --address 0.0.0.0 -n monitoring svc/grafana 8080:3000
    

    Important

    If you are running rok-tools as a Kubernetes StatefulSet you will need a double kubectl port-forward to also forward traffic from your local machine to the rok-tools Kubernetes Service:

    $ kubectl port-forward svc/rok-tools 3000:8080
    
  2. View the Grafana UI at localhost:3000 with your browser.

  3. Select any of the available dashboards from the upper-left dropdown list, next to the Home button.

    ../../../_images/grafana_ui.png

    Note

    It might take some time before metrics become available in dashboard panels. This depends on the scrape interval of each Prometheus target.