Free Node Space Monitoring¶
One of the most important areas to monitor in your EKF cluster is the availability of free space per node. This is essential to prevent hosts from running out of space, e.g., when workloads aggressively consume storage or fail to garbage collect old artifacts.
Before proceeding, ensure that you have been granted proper rights to access the Rok Monitoring Stack UI. Currently, access to the Rok Monitoring Stack is allowed only to admin users.
The Prometheus Node Exporter exposes metrics related to the available
filesystem space per node under the
The table below lists the Prometheus Node Exporter metrics that help you monitor the free space per physical node:
|node_filesystem_avail_bytes||Filesystem space available to non-root users in bytes||Gauge|
|node_filesystem_size_bytes||Filesystem size in bytes||Gauge|
|node_filesystem_free_bytes||Filesystem free space in bytes||Gauge|
|node_filesystem_files||Filesystem total file nodes||Gauge|
|node_filesystem_files_free||Filesystem total free file nodes||Gauge|
The Rok Monitoring Stack provides the following dashboards to visualize free space per node:
- Node Exporter: a full-fledged dashboard that queries and visualizes the majority of the metrics that the Prometheus Node Exporter collects.
- Nodes: a stripped-down version of the Node Exporter dashboard that queries and visualizes a subset with some of the most important metrics that the Prometheus Node Exporter collects.
- USE Method / Node: a concise dashboard targeted on node utilization, saturation, and errors.
- USE Method / Cluster: a concise dashboard targeted on cluster utilization, saturation, and errors.
USE Method stands for Utilization Saturation and Errors Method. The USE Method is a methodology for analyzing the performance of a system.
The Rok Monitoring Stack places Grafana dashboards for individual EKF
components under the
Visit the Kubeflow central dashboard with your browser athttps://<FQDN>
<FQDN>with your the value of your domain. For example:https://arrikto-cluster.apps.example.com
If prompted, log in using your credentials:
Select Metrics from the left side bar to navigate to Grafana:
In the left side bar, hover your cursor over the Dashboards entry and then click Manage to navigate to the Grafana Dashboards page:
In the Grafana Dashboards page you can search, view, and select dashboards.
Choose one of the following options, based on your needs and preferences:
In this guide you gained insight on how to monitor the available space of your EKF cluster nodes with the Rok Monitoring Stack.