Objective:
In our previous blog we have showcased how to authenticate using LDAP. But there are customers using open source Spinnaker for production usage and want even more security. And we help them securing their application even more, with the help of mutual TLS (mTLS) as it is based on certificates. mTLS is provides additional security because application requires private keys to be shared by two parties during communication.
In this blog, we shall show you how to share certificates between Spinnaker and LDAP servers. In other words, how to install LDAP with mutual TLS. Please be noted that if we set up MTLS for one Spinnaker instance, it will be used for that particular Spinnaker instance only.
Prerequisites for setting up MTLS in Spinnaker
- Spinnaker must be installed in Kubernetes cluster.
- Helm 3 Charts: By using HELM 3 charts, we are installing LDAP, and later we will share the certificates between LDAP and Spinnaker
In order to produce certs for Spinnaker and LDAP server, follow the six steps:
1. Installing cert-manager
Step 1.A
Run the command to fetch cert-manager from Github of cert-manager and apply into Kubernetes cluster
kubectl apply –validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.16.1/cert-manager.yaml
For Kubernetes version less than 1.15 we need follow below steps:
kubectl apply –validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.16.1/cert-manager-legacy.yaml
Alternative to Step-A: If you are using HELM repo, then you follow the below steps:
Step 1.B:
Cert-manager installation in the created namespace in Kubernetes ( in our case it is cert-manager)
( point to note: In a cluster, there can be multiple namespaces. And you can have you Spinnaker installed in one namespace and can have certificated installed in same/another namespace)
helm repo add jetstack https://charts.jetstack.io
helm repo add stable https://kubernetes-charts.storage.googleapis.com
helm repo update
kubectl create ns cert-manager
helm install certman --namespace cert-manager jetstack/cert-manager
2. Configuring issuer:
Objective: We are using self-signed certificates to get our certs to be trusted by both the parties ( Spinnaker and LDAP)
Step 2.A:
Create a yaml file with the content and apply to the Kubernetes cluster
File name: clusterissuer.yml
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: selfsigned-issuer
spec:
selfSigned: {}
Step 2.B:
Applying the created yaml with the content to K8S cluster
kubectl create -f clusterissuer.yml
kubectl get clusterissuer
Output should look something like the below:
NAME READY AGE
selfsigned-issuer True 10s
3. Configuring LDAP in Kubernetes Cluster:
Step 3.A:
Creating namespace by using the below command:
kubectl create ns <<LDAP-NAMESPACE>>
Output should look something like the below:
namespace/ldap created
Step 3.B:
Installing cert in LDAP namespace
StepB1: Create cacert.yml to get the certificate generated.
File name: cacert.yml (edit commonName & dnsNames)
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
name: hello-ca-tls
namespace: <<LDAP-NAMESPACE>>
spec:
# name of the tls secret to store the generated certificate/key pair
secretName: cacert
isCA: true
issuerRef:
# issuer created in step 1
name: selfsigned-issuer
kind: ClusterIssuer
commonName: "<<LDAP-NAMESPACE>>.svc.cluster.local"
dnsNames:
# one or more fully-qualified domain name can be defined here
- ldap-openldap.<<LDAP-NAMESPACE>>.svc.cluster.local
- localhost
Applying the configured yaml file in the LDAP namespace
kubectl create -f cacert.yml
kubectl -n <<LDAP-NAMESPACE>> get cert
Output should look something like the below:
NAME READY SECRET AGE
hello-ca-tls True cacert 23s
Step B2: To know the secret created or not we can use the following code
kubectl -n <<LDAP-NAMESPACE>> get secret
Output should look something like the below:
NAME TYPE DATA AGE
cacert kubernetes.io/tls 3 50s
Step 3.C:
Setup the issuer to get our certs to be trusted one.
Step 3.C.1:
Create a yaml file to create an issuer to produce authentication verification to our generated certs in the previous stage.
caissuer.yml
apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
name: caissuer
namespace: <<LDAP-NAMESPACE>>
spec:
ca:
secretName: cacert
To apply issuer use the following command
kubectl -n <<LDAP-NAMESPACE>> create -f caissuer.yml
kubectl -n <<LDAP-NAMESPACE>> get issuer
Output should look something like the below:
NAME READY AGE
caissuer True 17s
Step 3.D:
Create LDAP mtls certificates using the below files and apply in the kubernetes namespace ( where we will install LDAP)
ldapmutual.yml (Edit commonName and dnsNames)
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
Metadata:
name: ldapmutual
namespace: <<LDAP-NAMESPACE>>
spec:
secretName: ldapmutual
duration: 2160h # 90d
renewBefore: 360h # 15d
commonName: ldap-openldap.<<LDAP-NAMESPACE>>.svc.cluster.local
dnsNames:
- ldap-openldap.<<LDAP-NAMESPACE>>.svc.cluster.local
- localhost
- '*.<<LDAP-NAMESPACE>>.svc.cluster.local'
usages:
- digital signature
- key encipherment
- server auth
- client auth
issuerRef:
name: caissuer
# We can reference ClusterIssuers by changing the kind here.
# The default value is Issuer (i.e. a locally namespaced Issuer)
kind: Issuer
Apply the below commands to create mtls certs
kubectl create -f ldapmutual.yml
kubectl -n <<LDAP-NAMESPACE>> get certs
Output should look something like the below:
NAME READY SECRET AGE
hello-ca-tls True cacert 11m
ldapmutual True ldapmutual 14s
Check if the secrets are created or not:
kubectl -n ldap get secrets
NAME TYPE DATA AGE
cacert kubernetes.io/tls 3 12m
default-token-d5cjz kubernetes.io/service-account-token 3 16m
ldapmutual kubernetes.io/tls 3 53s
4. Install LDAP
In the Values.yml file we can override the default values ( like username, passwords, features) by LDAP HELM Charts
You can quickly download the file from the below link: https://github.com/helm/charts/blob/master/stable/openldap/values.yaml
Note: Please do enable the tls-enabled as true in the values.yaml
Step 4.A:
Use the below command to install LDAP
helm -n <<LDAP-NAMESPACE>> install ldap stable/openldap -f values.yml
Step 4.A.1:
Now check if the Kubernetes pods and services are up or not
kubectl -n <<LDAP-NAMESPACE>> get pod,svc
kubectl -n <<LDAP-NAMESPACE>> get pvc
Please note: The pod and pvc creation may take 5 or 6 minutes.
Step 4.A.2:
You can understand the details of the pod creation use the following command ( and yes you can certainly ignore the warning messages)
kubectl -n <<LDAP-NAMESPACE>> describe pod
Step 4A3: Connect into the LDAP pod to check LDAP connection
kubectl -n <<LDAP-NAMESPACE>> exec -it <podname> bash
ldapsearch -H ldap://localhost:389 -x -D "cn=admin,dc=example,dc=org" -w $LDAP_ADMIN_PASSWORD -b "dc=example,dc=org" dn memberof
5. Verify connection using tls
Follow the step to check if LDAP is connected with tls or not
Step 5.A:
Fetch the secret created (ldapmutual) in the previous steps
kubectl -n <<LDAP-NAMESPACE>> get secret ldapmutua
There will be 3 files ca.crt, tls.crt, and tls.key. Copy those certs from the secret ldapmutual into the client at /tmp directory or local machine (for me it is /decrypted-secrets/)
Note: Use “base 64 decode” for ca.crt, tls.crt and tls.key to decode the files
cat /tmp/ca.crt | base64 –decode >> /decrypted-secrets/ca.crt
cat /tmp/tls.crt | base64 –decode >> /decrypted-secrets/tls.crt
cat /tmp/tls.key | base64 –decode >> /decrypted-secrets/tls.key
cd /decrypted-secrets
Below command will create a sharable pkcs12 type file with encrypted secrets. ( here we have mentioned the output file name to be tls.p12)
openssl pkcs12 -export -clcerts -in tls.crt -inkey tls.key -out tls.p12 -name tls
Output will look like below ( you can use any password)
Enter Export Password:
Verifying – Enter Export Password:
To check if the file tls.p12 is created or not
ls –ltra
6. Copy the certs from the Spinnaker services from Gate microservice
Step 6.A
Copy the file “/etc/ssl/certs/java/cacerts “ from Gate pod of Spinnaker into the local directory (/decrypted-secrets)
Note: Here the path /etc/ssl/certs/java/cacerts might be different please use the respective path by checking the path in the gate pod.
Step 6.B
Import the secrets (ca.crt) which we have got from LDAP certs
keytool -import -alias custom-ca -keystore cacerts -file ca.crt
Step 6.C
Now let us create tls.jks file to place in the Spinnaker services
keytool -importkeystore -srckeystore tls.p12 -srcstoretype pkcs12 -destkeystore tls.jks -deststoretype pkcs12
Step 6.D
Till now we have created two certs: one from Gate pod in Spinnaker and another from the LDAP. By using those two we are creating secrets.
Below is the command to create a secret from the two certs
kubectl -n <<namespace-spinnaker>> create secret generic ldaptls --from-file tls.jks --from-file cacerts
Step 6.E
Connecting to the halyard pod to place our encrypted secret created in the above command.
kubectl exec -it <<halayard-pod>> bash -n <<spinnaker-namespace>>
In the halyard pod, we are modifying changes to direct the Spinnaker service to use our certs.
File path: ~/.hal/default/service-settings/gate.yml
healthEndpoint: /health
kubernetes:
useExecHealthCheck: false
volumes:
- id: ldaptls
mountPath: /etc/ssl/certs/java
type: secret
readOnly: true
env:
JAVA_OPTS: "-Djdk.tls.client.protocols=TLSv1.2 -Djavax.net.debug=all -Djavax.net.ssl.trustStore=/etc/ssl/certs/java/cacerts -Djavax.net.ssl.trustStorePassword=changeit -Djavax.net.ssl.keyStoreType=jks -Djavax.net.ssl.keyStore=/etc/ssl/certs/java/tls.jks -Djavax.net.ssl.keyStorePassword=changeit"
And similarly ~/.hal/default/service-settings/fiat.yml
kubernetes:
volumes:
- id: ldaptls
mountPath: /etc/ssl/certs/java
type: secret
readOnly: true
env:
JAVA_OPTS: "-Djdk.tls.client.protocols=TLSv1.2 -Djavax.net.debug=all -Djavax.net.ssl.trustStore=/etc/ssl/certs/java/cacerts -Djavax.net.ssl.trustStorePassword=changeit -Djavax.net.ssl.keyStoreType=jks -Djavax.net.ssl.keyStore=/etc/ssl/certs/java/tls.jks -Djavax.net.ssl.keyStorePassword=changeit"
Step 6.F:
So if LDAP is connected with mtls or not , we have to do the following changes.
Go to: ~/.hal/config
- Change the ldap url the following:
ldaps://<ldap svc name>.<ldap namespace>.svc.cluster.local:636/dc=example,dc=org
Go to:
~/.hal/default/profiles/fiat-local.yml
auth:
groupMembership:
service: ldap
ldap:
url: ldaps://<ldap svc name>.<<LDAP-NAMESPACE>>.svc.cluster.local:636
managerDn: cn=admin,dc=example,dc=org
managerPassword: <<LDAP_ADMIN_PASSWORD>> (PRESENT IN VALUES.YAML)
groupSearchBase: ou=groups,dc=example,dc=org
groupSearchFilter: member={0}
groupRoleAttributes: cn
userDnPattern: cn={0},dc=example,dc=org
And apply your changes
hal deploy apply
Now try to access the Spinnaker from UI
Conclusion:
By following the above mentioned steps, we can transfer data or information securely between LDAP and Spinnaker using mutual TLS certificates, thereby eliminating any security vulnerability risks.
References
https://www.openldap.org/doc/admin22/tls.html
https://github.com/helm/charts/tree/master/stable/openldap
https://cert-manager.io/docs/installation/kubernetes/
https://www.ibm.com/support/knowledgecenter/en/SSBS6K_3.1.2/manage_applications/create_issuer.html
0 Comments