In large-scale Kubernetes environments, Spinnaker’s components, particularly Clouddriver and Front50, work together to manage the lifecycle of applications and resources. However, a peculiar issue arises when synthetic applications—applications not created by any user—appear in the system. These “synthetic applications” are unknown to Front50 yet appear in the Spinnaker UI. This blog will describe the nature of synthetic applications, their possible hazards, and how to manage and prevent them.
What are the Synthetic Applications?
Synthetic apps are those that are built using Clouddriver without the explicit creation of any user in Front50. These apps show alongside ordinary applications in the Spinnaker UI since Clouddriver displays them while caching, but Front50 does not recognize them.
How Synthetic Applications Are Created?
There are two main methods that synthetic apps are created:
1. Moniker annotations in Kubernetes manifests:
When the annotation “moniker.spinnaker.io/application” appears in a Kubernetes manifest with a non-existent application name, Clouddriver automatically builds an application with that value. This happens regardless of how the manifest is deployed (via a “Deploy Manifest” stage, “Create Server Group” option, etc.). If the moniker is not set manually, Clouddriver defaults to the Spinnaker application name.
2. External Resources Managed Outside of Spinnaker:
When a Kubernetes account has onlySpinnakerManaged: false, Clouddriver creates applications for certain kinds of resources created outside of Spinnaker (without the Spinnaker moniker annotations). These resources typically include Deployments, DaemonSets, ReplicaSets, Jobs, StatefulSets, and CronJobs.
The first part of the resource’s name becomes the synthetic application name.
Observations on Synthetic Applications:
- Clouddriver’s Role: Synthetic applications exist only in Clouddriver and not in Front50, but they still show up in the Spinnaker UI as regular applications.
- Application Types Affected: Synthetic apps are primarily designed for Kubernetes resources listed in the “Clusters” section. Even with the namesake annotation, manifests such as Service or Pod do not result in the construction of synthetic applications.
- Application Renaming: When a moniker annotation is changed to a new non-existent application name, the synthetic application is renamed accordingly. This can cause confusion if the moniker is repeatedly changed.
- Pipelines in Synthetic Applications: Spinnaker allows users to create pipelines under these synthetic applications, which poses a risk if the underlying resource is deleted or altered.
Potential Issues with Synthetic Applications
While synthetic applications may seem harmless at first, they can lead to several problems:
- Pipeline Disruption: If pipelines are created under synthetic applications, they may become invisible if the resource (e.g., a Kubernetes deployment) that caused Clouddriver to create the synthetic application is deleted or updated. This may disrupt operations and complicate debugging.
- Security Gaps: Because synthetic apps do not exist in Front50, typical permissions for traditional programs are not enforced. This can cause security difficulties since unauthorized users may interact with these apps through pipelines.
- UI Inconsistencies: Because synthetic apps are presented in the Spinnaker UI, users who are unaware of their existence or purpose may become confused.
Handling Synthetic Applications
To avoid the risks associated with synthetic applications, several measures were taken for handling them across different Spinnaker components :
1. In Clouddriver: Validate Applications Before Caching Resources
Clouddriver’s caching agents need to ensure that any application created from a Kubernetes manifest is valid in Front50 before caching the associated resources. A configuration is in place to check that the application exists in Front50, preventing synthetic applications from being displayed in the UI.
Here is the configuration: kubernetes.checkApplicationInFront50: true # defaults to false.
2. Orca: Validate Pipeline Applications
Server group, cluster, and manifest pipeline steps in Orca should check against Front50 and Clouddriver applications to prevent Clouddriver from creating synthetic applications. If Front50 doesn’t have an application, Orca should throw an exception and stop pipeline execution.
This validation prohibits synthetic programs from establishing or running pipelines, preserving system integrity. This has been implemented in Orca by introducing a new task to check if the application exists in front50 for the pipeline stages of type Server Group, Cluster and Manifests. The following configuration enables this feature at individual stage level:
Separate config knobs are also provided at the AbstractCheckIfApplicationExistsTask level to determine if clouddriver needs to be queried for the application or not. It is by default set to true, so it is an opt-out capability. the config property is:
3. In Gate: Accept Front50 as Source of Truth for Applications.
In Gate, the front-end gateway service for Spinnaker, a rule can be enforced to treat Front50 as the sole source of truth for applications. By doing this, synthetic applications would be kept out of the UI entirely, preventing users from interacting with them.
While this won’t stop synthetic applications from being created via API calls, it ensures that they remain hidden from the user interface and cannot disrupt pipelines.
Though clouddriver is equipped with discarding caching manifests whose application name(derived from moniker) doesn’t exist in front50 but in spite of this, synthetic applications can exist in the following scenarios:
- When kubernetes.checkApplicationInFront50 is not enabled in clouddriver
- If any resources are already cached (and created synthetic applications) before enabling kubernetes.checkApplicationInFront50 in clouddriver.
The following property in Gate, which when enabled discards all the applications that are obtained from clouddriver which are not known to front50.
4. Notifying Synthetic App Development
It’s important to monitor the creation of synthetic applications. Setting up alerts to notify administrators when synthetic applications are created can help detect misconfigurations or unauthorized activity early.
Conclusion
Synthetic applications pretend to be harmless but may disrupt pipelines, compromise security, and cause UI issues in Spinnaker deployments. By adding checks across Clouddriver, Orca, and Gate, as well as considering Front50 as the source of truth, teams can verify that only legitimate applications are developed and maintained within Spinnaker.
Spinnaker customers may utilize these tactics to keep their applications and pipelines running well while avoiding the hazards associated with synthetic apps.
0 Comments