Prometheus, an open-source monitoring, and alerting toolkit is becoming the most popular solution for monitoring Kubernetes cluster for Spinnaker users or otherwise. To perform manual or automated Red/Black or Canary analysis, having a useful monitoring solution is essential.
In this blog, we will provide instructions on how to enable Prometheus monitoring for your Kubernetes cluster
Prometheus is by default a “pull-based” metrics system. The monitoring node needs to expose the metrics on “/metrics” endpoint (http://IP:9090/metrics), and the Prometheus server accesses the node’s “/metrics” endpoint on regular intervals (jobs) to collect the metrics. Node’s endpoint URLs should be configured on the Prometheus server for the server to access the nodes.
Enabling Prometheus Exporter on the Worker Nodes
sudo docker run \ --volume=/:/rootfs:ro \ --volume=/var/run:/var/run:rw \ --volume=/sys:/sys:ro \ --volume=/var/lib/docker/:/var/lib/docker:ro \ --volume=/dev/disk/:/dev/disk:ro \ --publish=9090:8080 \ --detach=true \ --name=cadvisor \ --restart always \ google/cadvisor:latest
The above command will launch cAdvisor container in the node and exposes metrics on http:<NODEIP>:9090/metrics. Run the above command on all worker nodes.
Next step is to configure the node IPs in the Prometheus server to allow it to pull the POD metrics off the nodes.
Enabling Prometheus Server
Step 1: Download Prometheus server from https://prometheus.io/download/
Step 2: Extract the .tar file and configure “prometheus.yml”, refer and add below configuration with the node IPs (where the cAdvisor containers were enabled in the previous section) under “scrape_configs:” section.
scrape_configs: - job_name: 'K8sCluster' honor_labels: true static_configs: - targets: ['NODE1 IP:9090'] - targets: ['NODE2 IP:9090'] - targets: ['NODE3 IP:9090']
Step 3: Start Prometheus server
nohup ./prometheus config.file=prometheus.yml &
Step 4: Check status of exporters(Monitoring nodes) in “9090/target” endpoint
Step 5: Check relevant metrics data
That’s it, you are done. For more advanced configurations, check out the complete documentation at https://prometheus.io/docs/introduction/overview/