What is APM ?
APM tools are used to monitor and track the overall health, performance, and behavior of the deployed app and the environment where it runs. For applications and microservices running in Kubernetes, Prometheus is a popular APM tool that provides detailed actionable metrics for the DevOps teams on the performance of containers, pods and services, and underlying Kubernetes cluster infrastructure. Prometheus server can be configured to trigger alerts to the Prometheus Alertmanager service instance, which then notifies the end-users through Email, Slack, or other communication channels.
Constraints with Prometheus as APM tool.
Although Prometheus and Alertmanager are effective monitoring tools to track threshold breaches, yet they do not support any type of authentication mechanism for connections to browser and HTTP API calls. This means anybody with the URL can access the monitoring information. However, enterprises need to have authentication to prevent unauthorized access to the monitoring information about their application deployments.
So if you are already using Prometheus or are considering one, then you can apply an authentication mechanism at the proxy layer, using any reverse proxy techniques. For a reverse proxy to enforce basic authentication, you can use NGINX Ingress Controller , which is a traffic management solution for cloud‑native apps in Kubernetes containerized environments.
After deploying applications on the Kubernetes platform using Spinnaker we need to use APM (application performance monitoring) tools in a secured way using Authentication.
This blog explains how to implement the Basic Authentication for both Prometheus and Alertmanager running on the Kubernetes using NGINX as the reverse proxy Ingress Controller.
This blog assumes that you have the Prometheus and Alertmanager installed on your Kubernetes cluster and both of them are accessible in the web browser without any authentication mechanism. If not installed, please refer to Prometheus Installation using the steps mentioned here.
- Creating Basic Authentication credentials.
- Creating Kubernetes Secret from the Basic Auth File.
- Creating/ Updating the Ingress rule with the Basic Authentication file.
- Testing your Ingress.
Create Basic Authentication credentials for securely accessing APM tools used in a Spinnaker CD pipeline:
Basic Authentication credentials can be created using the htpasswd tool. The htpasswd tool creates and updates the flat-files used to store usernames and passwords for basic authentication of the HTTP users. Given below is the example to create the basic auth file:
$ htpasswd -c auth-details
Retype new password:
Adding password for user
After you enter a username and password, a new auth file will be created with the name “auth-details”.
Creating Kubernetes Secret from the Basic Auth File:
Once you have the basic auth file created, now Kubernetes Secret can be created using the below command,
$ kubectl create secret generic basic-auth --from-file=auth-details
As the Secret gets created successfully, you can see the Secret using the below command:
$ kubectl get secret basic-auth -o yaml \\The sample output is shown as: apiVersion: v1 data: auth: Zm9vOiRhcHIxJE9GRzNYeWJwJGNrTDBGSERBa29YWUlsSDkuY3lzVDAK kind: Secret metadata: name: basic-auth namespace: default type: Opaque
Creating/ Updating the Ingress rule with the Basic Authentication file:
To enable the Basic Authentication at the Kubernetes Ingress layer we need to add the below annotations to the Kubernetes Nginx Ingress resource:
nginx.ingress.kubernetes.io/auth-type: basic \\(This indicates the type of Authentication.) nginx.ingress.kubernetes.io/auth-secret: basic-auth \\(Name of the secret that contains the user/password definitions.) nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required - ' \\(Message to display with an appropriate context of why the authentication is required.)
Now set the configuration of the following Ingress config files enabling Basic Authentication:
apiVersion: extensions/v1beta1 kind: Ingress metadata: annotations: cert-manager.io/issuer: letsencrypt-prod kubernetes.io/ingress.class: nginx nginx.ingress.kubernetes.io/auth-realm: Authentication Required - admin nginx.ingress.kubernetes.io/auth-secret: basic-auth nginx.ingress.kubernetes.io/auth-type: basic creationTimestamp: "2020-11-23T13:32:25Z" generation: 1 name: prometheus-ingress namespace: oes resourceVersion: "33407088" selfLink: /apis/extensions/v1beta1/namespaces/oes/ingresses/prometheus-ingress uid: faeb0983-382a-4172-8a30-a0cd9dfc5b13 spec: rules: - host: demoprom.opsmx.com http: paths: - backend: serviceName: prom-prometheus-server servicePort: 80 path: / tls: - hosts: - demoprom.opsmx.com secretName: promtls status: loadBalancer: ingress: - ip: 10.240.0.6
apiVersion: extensions/v1beta1 kind: Ingress metadata: annotations: cert-manager.io/issuer: letsencrypt-alertmanager kubernetes.io/ingress.class: nginx nginx.ingress.kubernetes.io/auth-realm: Authentication Required - admin nginx.ingress.kubernetes.io/auth-secret: basic-auth nginx.ingress.kubernetes.io/auth-type: basic creationTimestamp: "2020-11-24T04:32:56Z" generation: 1 name: alertmanager-ingress namespace: oes resourceVersion: "33561571" selfLink: /apis/extensions/v1beta1/namespaces/oes/ingresses/alertmanager-ingress uid: 4e1150dc-34cf-4854-8b7c-e85a31737283 spec: rules: - host: demoalerts.opsmx.com http: paths: - backend: serviceName: prom-prometheus-alertmanager servicePort: 80 path: / tls: - hosts: - demoalerts.opsmx.com secretName: alerttls status: loadBalancer: ingress: - ip: 10.240.0.6
Testing your Ingress:
Use cURL to interact with your NGINX/ Prometheus setup. Try this request:
$ curl https://demoprom.opsmx.com
If you try to reach the Prometheus it will show the following 401 authentication error:
401 Authorization Required
For successful authentication, you need to pass the username with the -u flag and enter the password when prompted.
$ curl https://demoprom.opsmx.com -u admin
Enter host password for user ‘admin’:
After you enter the password, the following output is shown signaling that the authentication has been successful.
In this blog we have shown you how to store a username and password in an auth file and create Kubernetes secret from it, configured the NGINX Ingress layer for reverse proxying and use the auth file credentials to authenticate users accessing Prometheus and Alertmanager HTTP endpoints. Now you should be able to enhance security in monitoring your new releases.
If you want to know more about the Spinnaker or request a demonstration, please book a meeting with us.
OpsMx is a leading provider of Continuous Delivery solutions that help enterprises safely deliver software at scale and without any human intervention. We help engineering teams take the risk and manual effort out of releasing innovations at the speed of modern business. For additional information, contact us