Spinnaker has multiple components ( or microservices) like Orca, Igor, Clouddriver, and so on. Each component has a specific role. While deploying or installing Spinnaker in Kubernetes, each component is installed separately.
Enterprises often need to use custom Docker container images to specify their project-specific dependencies and incorporate additional functionality, and features that are not part of the open-source Spinnaker. But while using custom images to install Spinnaker, often a common issue faced is that the pods of that installed Spinnaker service fail to execute and run, as the “readiness probe” (Kubernetes health check probe for pods) fails.
Here in this blog, we will discuss how OpsMx has found a way to resolve this issue and get you going with your Spinnaker deployments.
When we use a custom image for microservice in Spinnaker, the pods of that service will not come up as the readiness probe fails since the custom image may not have the tools like netstat, wget, or curl.
Here, we have a custom docker container image for the Spinnaker’s Orca microservice for which the readiness probe fails.
For e.g. :
Custom image used:
artifactId: docker.io/opsmxdev/orca:1.22.2-pr3989-40 kubectl describe po spin-orca-fff44f5f4-8j6zs
Readiness probe failed
Steps for resolution
The following steps will help you in resolving the issue as discussed above you can use and implement this resolution very easily. Execute the steps as given below:
1. Manually remove readiness probe from that service’s deployment as below:
kubectl edit deploy spin-orca
readinessProbe: exec: command: - wget - --no-check-certificate - --spider --q - https://localhost:8083/health failureThreshold: 3 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1
2. Verify the new pod that comes up with the custom image.
kubectl describe po spin-orca-fff44f5f4-8j6ab orca: Container ID: docker:// 05bdc0f4122ff67ef928497cfdebc4ac78de86763fc36ba6855a64b9be0e55f2 Image: docker.io/opsmxdev/orca:1.22.2-pr3989-40
3. Now execute the custom image in the new pod.
kubectl -n spinnaker exec -it spin-orca-86df845bf6-c7sk6 -- bash
4. Verify the ‘wget’ tool used for the health check probe.
wget Output: wget: missing URL
5. Do the health check manually in the pod.
wget --no-check-certificate --spider -q --private-key /pkcs12/tls.key --certificate /pkcs12/tls.crt https://localhost:8083/health
6. Change the health check command accordingly in the service.yaml
artifactId: docker.io/opsmxdev/orca:1.22.2-pr3989-40 scheme: https kubernetes: useExecHealthCheck: true customHealthCheckExecCommands: ["wget", "--no-check-certificate", "--spider", "-q", "--private-key", "/pkcs12/tls.key", "--certificate", "/pkcs12/tls.crt", "https://localhost:8083/health"] overrideBaseUrl: https://spin-orca.devmt.isym.io:8083 healthEndpoint:
7. Deploy Halyard
hal deploy apply
Note: Do not use –service-names=orca, as the monitoring daemon. It is causing issues with individual updates.
OpsMx offers Intelligent Software Delivery (ISD) for Spinnaker to accelerate approvals, verification, and compliance checks by analyzing data generated across Spinnaker pipelines and the software delivery chain. ISD for Spinnaker is available as SaaS, managed service, or on-premises.
Founded with the vision of “delivering software without human intervention,” OpsMx enables customers to transform and automate their software delivery processes. OpsMx builds on open-source Spinnaker and Argo with services and software that helps DevOps teams SHIP BETTER SOFTWARE FASTER.