Skip to content

Configuration Drift detection

Configuration Drift

Configuration drift is a commonly used term to describe a change that takes place in an environment. Drift is an issue as it causes systems and parts of a system which supposed to be consistent, to become inconsistent and unpredictable. In our case, configuration drift is a change of a resource deployed by Sveltos down the managed clusters.

Sveltos allows users to set the sync mode within a ClusterProfile to ContinuousWithDriftDetection. It enables Sveltos to monitor the state of managed clusters and detect configuration drift for any of the resources deployed by a ClusterProfile.

---
apiVersion: config.projectsveltos.io/v1beta1
kind: ClusterProfile
metadata:
  name: deploy-kyverno
spec:
  syncMode: ContinuousWithDriftDetection
  helmCharts:
  - repositoryURL:    https://kyverno.github.io/kyverno/
    repositoryName:   kyverno
    chartName:        kyverno/kyverno
    chartVersion:     v3.1.4
    releaseName:      kyverno-latest
    releaseNamespace: kyverno
    helmChartAction:  Install

When Sveltos detects a configuration drift, it will automatically re-sync the cluster state back to its original state which is described in the management cluster. Sveltos deploys a service in each managed cluster and configures it with a list of Kubernetes resources deployed for each ClusterProfile in SyncModeContinuousWithDriftDetection mode.

The service starts a watcher for each GroupVersionKind with at least one resource to watch. When any watched resources are modified (labels, annotations, spec or rules sections), the service notifies the management cluster about potential configuration drifts. The management cluster then reacts by redeploying affected ClusterProfiles.

This way, Sveltos ensures that the systems are always consistent and predictable, preventing unexpected issues caused by the configuration drifts.

Configuration drift recovery

Ignore Annotation

You can stop certain resources from being tracked for configuration drift. This is done by adding a special annotation projectsveltos.io/driftDetectionIgnore to those resources.

For instance, following ClusterProfile will deploy Kyverno helm chart. Patches are used to apply annotation to the Kyverno kyverno-admission-controller deployment. This means any changes made to resources deployed by the Helm chart itself will be flagged as a configuration drift. However, any modifications directly to the kyverno-admission-controller deployment won't be detected as drift.

    apiVersion: config.projectsveltos.io/v1beta1
    kind: ClusterProfile
    metadata:
      name: deploy-kyverno
    spec:
      clusterSelector:
        matchLabels:
          env: fv
      syncMode: ContinuousWithDriftDetection
      helmCharts:
      - repositoryURL:    https://kyverno.github.io/kyverno/
        repositoryName:   kyverno
        chartName:        kyverno/kyverno
        chartVersion:     v3.1.4
        releaseName:      kyverno-latest
        releaseNamespace: kyverno
        helmChartAction:  Install
      patches:
      - target:
          group: apps
          version: v1
          kind: Deployment
          name: "kyverno-admission-controller"
        patch: |
          - op: add
            path: /metadata/annotations/projectsveltos.io~1driftDetectionIgnore
            value: "ok"

Customize drift-detection-manager configuration

In some cases, you might want to tailor the deployment of the drift-detection-manager1. To achieve this, the addon-controller pod now accepts a new argument named drift-detection-config.

This argument points to a ConfigMap within the projectsveltos namespace. The ConfigMap holds patches that will be applied to the drift-detection-manager before its deployment in the managed cluster.

Here's an example:

apiVersion: v1
data:
  patch: |-
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: drift-detection-manager
    spec:
      template:
        spec:
          containers:
          - name: manager
            image: docker.io/projectsveltos/drift-detection-manager:dev
            resources:
              requests:
                memory: 256Mi
            securityContext:
              readOnlyRootFilesystem: true
kind: ConfigMap
metadata:
  name: drift-detection
  namespace: projectsveltos

Along with creating the ConfigMap, you'll also need to configure the addon-controller deployment to use it. To do this, add the following argument to the deployment:

- args:
  ...
  - --drift-detection-config=drift-detection

With this configuration, the drift-detection-manager will be deployed in each managed cluster with the following settings:

  • Request memory: 256Mi
  • Image: projectsveltos/drift-detection-manager:dev

  1. Same is valid for sveltos-agent. classifier pod now accepts a new argument named sveltos-agent-config. It points to a ConfigMap in the projectsveltos namespace. The ConfigMap holds patches that will be applied to the sveltos-agent before its deployment in the managed cluster. ↩