Background: Why to import manifest files from Github to Artifactory?
DevOps engineers and developers would be familiar with json or yaml files applied to ensure Kubernetes maintains the desired state of application/object. These yaml files are also called manifest files or referred to as Kubernetes API objects. The open source CD tool , Spinnaker, uses these manifest files in two ways- direct text format or in artifact format- to be able to deploy applications. Whenever artefacts from an external source are used, the URL has to be specified in the Spinnaker. These sources can be Github, Bitbucket, Artifactory, or Docker Hub, etc.
People are migrating from Git to Artifactory
Github, acquired by Microsoft, is one of the leading platforms used by developers to store their code, build, manage and control the same in their relevant projects. It offers Git as an open source version control system/software which enables multiple developers to commit changes at the same time. On the other hand, Artifactory is a repository manager, it helps developers seamlessly access artifacts that are present in remote repositories, supports various software packages, manages and distributes binary intelligently.
As enterprises are frequently releasing new versions of existing software/applications, these two tools by facilitating the aforementioned processes have become a quintessential part of Devops technology today and for our case both can serve as a resource to store kubernetes manifest files. Even though both provide CI/CD services, GIthub has far more outreach in companies and developing circles compared to Artifactory, however when it comes to SDLC management, devsecops, supporting various software packages, storing binaries, integration with various cloud native tools, vulnerabilities and licensing and many more such features Artifactory out performs Github.
Hence a lot of enterprises are adopting Artifactory in their production environments.
Scenarios for importing manifest files from Github
- Various enterprises today use different repository solutions for different environments- Dev, QA, Prod environments. For e.g. Github/Bitbucket could be used for dev and QA, and Artifactory for prod env.
- Certain enterprise decides to migrate from Github to Artifactory.
In both the cases above we will have to move our manifest files from Github that we have been using to deploy our Kubernetes applications. Manual migration is cumbersome ofcourse, hence I have used Jenkins to create a job and export the files from Github to Artifactory. One can obviously create job and trigger it based on any events or a specific time of a day, but I have explained a no-frill job in this blog.
To achieve the automatic import of manifest files from Git to Jfrog and deployment using Spinnaker is divided into two steps:
- Step-1 will tell you how to integrate Jenkins with Artifactory, create a Jenkins job to fetch manifests from Github to Artifactory
- Step-2 will tell you how to configure Spinnaker pipeline to fetch manifest files- stored in Artifactory- for deployments
- Jenkins Account
- Artifactory Account – Trial version used for this blog.
- Spinnaker Account
- GitHub Repository – Used Public repository however private repository can be used as well.
** Disclaimer ( use different ports): If any of you all are installing Spinnaker and Jenkins for the first time and doing so in the same system/VM, please ensure that the port you configure for Jenkins is 8081 to avoid conflict with Spinnaker as Fiat microservice in Spinnaker uses port 8080.
Step 1: Integrate Jenkins with Artifactory
Add a Artifactory plugin to Jenkins
To integrate Jenkins with Artifactory we have to check whether Artifactory plugin is installed or not, we can do so by simply going to Manage Jenkins-> Manage Plugins and check by clicking the installed tab:
In case, you do not have Artifactory Plugin installed simply go to Available tab and then on the top right hand corner in Filter type Artifactory, and then install the plugin.
After installation of the plugin we will go to Configure System option from Manage Jenkins scroll down and edit:
Server ID : Unique name to your server for eg: opsmx-art
URL : URL Link to your artifactory for eg: http://<artifactory server IP>:8081/artifactory
We also need to enter Default Deployer Credentials which is basically login credentials for your Artifactory.
Finally we will Test Connection and we should get a message stating Found Artifactory along with the version. Save your configuration.
Below is an image for the above settings:
Test the connection between Jenkins and Artifactory
At times after adding Default Deployer Credentials,then saving the configuration and later on coming back to Test Connection might throw errors shown in the below image:
To avoid such errors we need to simply tick the checkbox Use the Credentials Plugins and enter the relevant credentials, the Test Connection will work after saving your configuration.
Build a Jenkins Job to fetch artifact from GitHub and Push it to Artifactory
Creating Repositories in Artifactory
For the artifact we have used a public git repository, however we can always use private repositories as well, all we need to do is pass our credentials during the job setup.
Also as we have used Trial version of Artifactory we created some of the Generic and Maven Repositories manually, Enterprise versions of Artifactory usually come with more features and default repositories are available based on package types.
Below is an image of Package Types in Artifactory:
For our testing purpose we created some local repositories shown in below image, depending on the JFrog Artifactory Subscription you can create repositories or use the default ones.
To create repositories in Artifactory, you will have to go to Repositories option at top left hand corner,then select New Local Repository -> Select your Package type -> Enter Repository Key details -> Save & Finish.
Creating a Jenkins Job
For Jenkins Job we have to configure a Pipeline first, in our case we have configured a pipeline called opsmx-art-testing.
In the Source Code Management select Git and enter:
Repository URL: URLof the git repository which has the manifest files
Credentials: If its public repository credentials need not be added it is only for private repositories.
Branches Build: In our case we are building from /master however you can add any branch from which you want to build.
Next we will check Generic-Artifactory Integration checkbox in Build Environment.
Next steps are crucial, as we scroll below in the Build Environment tab we get to Artifactory Configuration.
We will start by configuring Upload Details:
Artifactory Upload Server: http://<artfactory server IP>:8081/artifactory
Target Repository: Mention the repository in Artifactory where you want the manifest files to be pushed. In my case I have used opsmx-foa-test as my repository.
Published Artifacts: This contains the structure of the folder/files to be fetched from GIt. This part can be tricky, please exercise caution while entering the structure.
Sharing Jenkins Configuration Image below:
Corresponding GitHub Repository Image:
** Note: Even though this repository was built for the Maven package we used Generic Artifactory Integration and it works.
Next we will add Artifactory download server details in Download Details:
Artifactory Download Server: http://<artfactory server IP>:8081/artifactory
Save the configuration.
Run and Test the Jenkins Job
We run the above Jenkins Job next. You will get the Artifactory Build Info option for the Jenkins job that you will configure.
Finally we will be able to see the manifest files that we fetched from Github in our Artifactory.
Now the second part of the blog where we configure our Spinnaker artifact that will use the manifest file in Artifactory.
Step 2: Configuring Spinnaker
We need to enable http in Spinnaker first. This can be done by going into your halyard pod and firing the below command:
## hal config artifact http account add ACCOUNT –username <username> –password
The above will have your Artifactory details. Below is a link to the command structure:
Once http is enabled we will configure our pipeline. The example below deploys nginx.
I have configured the pipeline in an application called ‘artint’. In the Configuration stage we will configure the Expected Artifacts option. We will enter Account and URL details in the Expected Artifacts , the url link is the complete path to your manifest file in Artifactory.
Add a Display name, in our case we named it as ‘nginx-yaml’, you can use your desired name based on the type of pipeline you are configuring. We will check the Use Default Artifact option and enter the Artifactory URL details.
Next we will configure Deploy stage:
You will have to check the Artifact option as Manifest Source, Manifest Artifact will have the Display Name you added in Configuration stage, you will have to select the same. In my case it is ‘nginx-yaml’.
We will save our configuration and execute the pipeline:
** Note, in the Expected Artifact section the URL that I have provided as artifact points to /opsmx-yaml/nginx.yaml, where opsmx-yaml is a folder, this is another repository that I had created this is not to be confused with the repository example used to show fetching of manifest files from Github to Artifactory where I had used opsmx-foa-test.
And with that I leave you all to explore these features, try out various combinations of folder structures and see if you are able to fetch them from Github and use them successfully to configure your Spinnaker pipeline.
Hope you have enjoyed the blog. In case you have any query please feel free to drop a message at [email protected]