11 software delivery problems solved by intelligence software delivery platform  Download
Select Page
Share

Prometheus, an open-source monitoring, and alerting toolkit is becoming the most popular solution for monitoring Kubernetes cluster for Spinnaker users or otherwise.   To perform manual or automated Red/Black or Canary analysis, having a useful monitoring solution is essential.

In this blog,  we will provide instructions on how to enable Prometheus monitoring for your Kubernetes cluster

Background:

Prometheus is by default a “pull-based” metrics system.  The monitoring node needs to expose the metrics on “/metrics” endpoint (http://IP:9090/metrics), and the Prometheus server accesses the node’s “/metrics” endpoint on regular intervals (jobs) to collect the metrics. Node’s endpoint URLs should be configured on the Prometheus server for the server to access the nodes.

Enabling Prometheus Exporter on the Worker Nodes

To enable Prometheus exporter, launch cAdvisor container on every worker node in the cluster. For info information on cAdvisor, check out  https://github.com/google/cadvisor

sudo docker run \

  --volume=/:/rootfs:ro \

  --volume=/var/run:/var/run:rw \

  --volume=/sys:/sys:ro \

  --volume=/var/lib/docker/:/var/lib/docker:ro \

  --volume=/dev/disk/:/dev/disk:ro \

  --publish=9090:8080 \

  --detach=true \

  --name=cadvisor \

  --restart always \

  google/cadvisor:latest

The above command will launch cAdvisor container in the node and exposes metrics on http:<NODEIP>:9090/metrics.  Run the above command on all worker nodes.

Next step is to configure the node IPs in the Prometheus server to allow it to pull the POD metrics off the nodes.

Enabling Prometheus Server

Step 1: Download Prometheus server from https://prometheus.io/download/

Step 2: Extract the .tar file and configure “prometheus.yml”, refer and add below configuration with the node IPs (where the cAdvisor containers were enabled in the previous section) under “scrape_configs:” section.

scrape_configs: 

   - job_name: 'K8sCluster'

     honor_labels: true

     static_configs:

       - targets: ['NODE1 IP:9090']

       - targets: ['NODE2 IP:9090']

       - targets: ['NODE3 IP:9090']

Step 3: Start Prometheus server

nohup ./prometheus config.file=prometheus.yml &

Step 4:  Check status of exporters(Monitoring nodes) in “9090/target” endpoint

Prometheus Targets

Prometheus Targets

Step 5:  Check relevant metrics data

Prometheus Metric Graph

Prometheus Metric Graph

That’s it, you are done.  For more advanced configurations, check out the complete documentation at https://prometheus.io/docs/introduction/overview/

You can connect this Prometheus server to your Automated Canary Analysis system to able to perform automated release analysis.  If you need further assistance, email [email protected]

Balaji Sivasubramanian

Balaji Sivasubramanian is the Vice President of Product at OpsMx. He is passionate about creating and evangelizing winning products with expertise in all aspects of the business – strategy, product management, product marketing, GTM, and business development in early-stage startups and innovative companies. He is a Subject Matter Expert in CI/CD, DevOps, Applications, Containers/Microservices, Data Center IT / Cloud Infrastructure - Network, Compute, and Storage, IT Security, Machine Learning and AI, Enterprise, SaaS, and Cloud. He is an ambassador for the Continuous Delivery Foundation (CDF).

6 Comments

  1. Alex

    I thought that you didn’t need to deploy cadvisor as kubelet integrates it directly?

    Reply
    • Balaji Sivasubramanian

      Alex, Thanks for your comment. You are correct that kublet has built-in cadvisor which is also used by Heapster for multinode monitoring. This blog was to demonstrate single node monitoring using cAdvisor as a Prometheus target.

      Reply
  2. Alex

    I thought that you didn’t need to deploy cadvisor as kubelet integrates it directly?

    Reply
    • Balaji Sivasubramanian

      Alex, Thanks for your comment. You are correct that kublet has built-in cadvisor which is also used by Heapster for multinode monitoring. This blog was to demonstrate single node monitoring using cAdvisor as a Prometheus target.

      Reply
  3. Alex

    I thought that you didn’t need to deploy cadvisor as kubelet integrates it directly?

    Reply
    • Balaji Sivasubramanian

      Alex, Thanks for your comment. You are correct that kublet has built-in cadvisor which is also used by Heapster for multinode monitoring. This blog was to demonstrate single node monitoring using cAdvisor as a Prometheus target.

      Reply

Trackbacks/Pingbacks

  1. Basic Authentication for Prometheus & Alertmanager in Spinnaker - […] This blog assumes that you have the Prometheus and Alertmanager installed on your Kubernetes cluster and both of them…

Submit a Comment

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.