Creating your first backup

In this guide, you will create a backup of a Kafka topic. The backup will be stored on a Persistent Volume .

You will configure:

• The EventHub to define the Kafka cluster.
• The Credentials to authenticate with the Kafka cluster.
• The Storage to store data on.
• The Backup using all of the above.

Pre-requisites

To follow this guide, make sure you have the following tools available:

Kubernetes

• A Kubernetes cluster with Kannika Armory installed.
• A running Kafka cluster
• The kubectl CLI tool
• Access to the Kubernetes API.

GraphQL

• A Kubernetes cluster with Kannika Armory installed.
• A running Kafka cluster
• A GraphQL client like Postman.
• Access to the GraphQL API.

REST

• A Kubernetes cluster with Kannika Armory installed.
• A running Kafka cluster
• An HTTP client like Postman.
• Access to the REST API.

Step 1: Configure an EventHub

An EventHub is a Kannika Armory resource that is used to depict an external system that contains event data. In this step, you will create a Kafka EventHub.

Kubernetes

Create the following resource in Kubernetes:

apiVersion: kannika.io/v1alpha
kind: EventHub
metadata:
  name: my-kafka-cluster
spec:
  kafka:
    properties:
      bootstrap.servers: broker:9092

GraphQL

mutation {
  configureKafkaEventHub(
    input: {
      name: "my-kafka-cluster",
      bootstrapServers: "broker:9092"
    }
  ) {
    ...on KafkaEventHubConfigured {
      eventHub {
        name
      }
    }
    ...on KafkaEventHubConfigurationFailed {
      problems {
        message
        type
      }
    }
  }
}

REST

POST /rest/eventhubs HTTP/1.1
Content-Type: application/json

{
  "name": "my-kafka-cluster",
  "kafka" : {
     "bootstrap_servers": "broker:9092"
  }
}

In this example:

• A Kafka EventHub named my-kafka-cluster is created.

• The EventHub points to a Kafka cluster at broker:9092.

---

To authenticate with the Kafka cluster, you’ll need to configure Credentials. In this example, you’ll use SASL/PLAIN Credentials with a username and password.

Kubernetes

Create the following resources in Kubernetes:

# Define the SASL Credentials
apiVersion: kannika.io/v1alpha
kind: Credentials
metadata:
  name: sasl-creds
spec:
  sasl:
    mechanism: PLAIN
    usernameFrom:
      secretKeyRef:
        name: sasl-creds  # References the Secret below
        key: username
    passwordFrom:
      secretKeyRef:
        name: sasl-creds  # References the Secret below
        key: password
    sslConf:
      enabled: true
---
# Define the Secret that contains the username and password
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: sasl-creds
data:
  username: c2FzbC11c2VybmFtZQ== # sasl-username, base64 encoded
  password: c2FzbC1wYXNzd29yZA=== # sasl-password, base64 encoded

GraphQL

mutation {
  configureSaslCredentials(
    input: {
      name: "sasl-creds",
      mechanism: PLAIN,
      username: "sasl-username",
      password: "sasl-password",
      useSsl: true
    }
  ) {
    ... on SaslCredentialsConfigured {
      credentials {
        name
      }
    }
    ... on SaslCredentialsConfigurationFailed {
      problems {
        message
        type
      }
    }

  }
}

REST

POST /rest/credentials HTTP/1.1
Content-Type: application/json

{
  "name": "sasl-creds",
  "sasl": {
    "mechanism": "PLAIN",
    "username": "sasl-username",
    "password": "sasl-password",
    "use_ssl": true
  }
}

In this example:

• The Credentials are used for SASL/PLAIN authentication with the username sasl-username and password sasl-password, which are stored in a Secret named sasl-creds.

• SSL encryption is enabled. This correlates to the Kafka property security.protocol being set to SASL_SSL.

Step 2: Configure a Storage

Next, configure the Storage which will be used to store the backup data. In this example, a Volume Storage is used, which will store the data on a Persistent Volume .

Kubernetes

Create the following Storage resource in Kubernetes:

apiVersion: kannika.io/v1alpha
kind: Storage
metadata:
  name: "my-volume-storage"
spec:
  volume:
    capacity: 10Gi

GraphQL

mutation {
  configureVolumeStorage(
    input: {
      name: "my-volume-storage",
      capacity: "10Gi"
    }
  ) {
    ... on VolumeStorageConfigured {
      storage {
        name
      }
    }
    ... on VolumeStorageConfigurationFailed {
      problems {
        message
        type
      }
    }
  }
}

REST

POST /rest/storages HTTP/1.1
Content-Type: application/json

{
  "name": "my-volume-storage",
  "volume": {
    "capacity": "10Gi"
  }
}

In this example:

• A Volume Storage named my-volume-storage is created.

• The volume has a capacity of 10Gi.

Step 3: Configure the backup

Finally, create a Backup which will back up data from the EventHub on the Storage.

Kubernetes

Create the following Backup resource in Kubernetes:

apiVersion: kannika.io/v1alpha
kind: Backup
metadata:
  name: "my-backup"
spec:
  source: "my-kafka-cluster"
  sourceCredentialsFrom:
    credentialsRef:
      name: "sasl-creds"
  sink: "my-volume-storage"
  enabled: true             # Optional, defaults to true
  streams:
    - topic: "my-topic"
      enabled: true         # Optional, defaults to true

GraphQL

Configure the backup using the configureBackup GraphQL mutation:

mutation {
  configureBackup(
    input: {
      name: "my-backup",
      sourceEventHubName: "my-kafka-cluster",
      sourceCredentialsName: "sasl-creds",
      sinkStorageName: "my-volume-storage",
    }
  ) {
    ...on BackupConfigured {
      backup {
        name
     }
     ...on BackupConfigurationFailed {
       problems {
         message
         type
       }
     }
  }
}

Then, import topics using the enableBackupStreams GraphQL mutation:

mutation {
  configureBackupTopics(
    input: {
      backupName: "my-backup",
      topics: ["my-topic"]
      enable: true
    }
  ) {
    ...on BackupTopicsConfigured {
      backup {
        name
      }
    }
    ...on ConfigureBackupTopicsFailed {
      problems {
        message
        type
      }
    }
  }
}

REST

First, create the backup using the POST /rest/backups endpoint:

POST /rest/backups HTTP/1.1
Content-Type: application/json
{
  "name": "my-backup",
  "source": "my-kafka-cluster",
  "source_credentials": "sasl-creds",
  "sink": "my-volume-storage",
}

Then, import topics using the POST /rest/backups/{backupName}/topics endpoint:

POST /rest/backups/my-backup/topics HTTP/1.1
Content-Type: application/json

{
  "topics": [
    "my-topic"
  ],
  "enable": true
}

In this example:

• The Backup is named my-backup.

• The Backup connects to the EventHub my-kafka-cluster.

• The Backup uses the Credentials sasl-creds to authenticate with the Kafka cluster.

• The Backup stores data on the Storage my-volume-storage.

• The topic my-topic is added to the Backup.

Step 4: Verify that the backup is running

Congratulations on making it this far! You have now created your first backup, and it should be running.

Kubernetes

To verify that the backup is running, run the following command:

$ kubectl get backups

You should see output similar to the following:

NAME        STATUS
my-backup   🚀 Streaming

You should also see the associated Pod running:

$ kubectl get pods

You should see output similar to the following:

NAME                         READY   STATUS    RESTARTS   AGE
my-backup-75465ff49-kcw8p    1/1     Running   0          1s

GraphQL

To verify that the backup is running, run the following query:

query {
  getBackups(first: 10) {
    edges {
      node {
        name
        uuid
        status
        paused
        config {
          sourceRef {
            name
            eventHub {
              ...on KafkaEventHub {
                name
                properties {
                  key
                  value
                }
              }
            }
          }
          sinkRef {
            name
          }
        }
      }
    }
  }
}

You should see output similar to this:

{
  "data": {
    "getBackups": {
      "edges": [
        {
          "node": {
            "name": "my-backup",
            "uuid": "40b31171-913e-4b95-9c77-ad71db981dc0",
            "status": "RUNNING",
            "paused": false,
            "config": {
              "sourceRef": {
                "name": "my-kafka-cluster",
                "eventHub": {
                  "name": "my-kafka-cluster",
                  "bootstrapServers": "broker:9092"
                  ]
                }
              },
              "sinkRef": {
                "name": "my-volume-storage",
              },
              "streams": [
                "topic": "my-topic"
              ]
            }
          }
        }
      ]
    }
  }
}

REST

To verify that the backup is running, run the following command:

GET /rest/backups HTTP/1.1

You should see output similar to this:

HTTP/1.1 200
Vary: Origin
Vary: Access-Control-Request-Method
Vary: Access-Control-Request-Headers
X-Content-Type-Options: nosniff
X-XSS-Protection: 0
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: 0
X-Frame-Options: DENY
Content-Type: application/json
Transfer-Encoding: chunked
Date: Thu, 30 May 2024 12:41:00 GMT

{
  "backups": [
    {
      "name": "my-backup",
      "uuid": "40b31171-913e-4b95-9c77-ad71db981dc0",
      "config": {
        "source": "my-kafka-cluster",
        "sink": "my-volume-storage",
        "topics": ["my-topic"],
        "source_credentials": "sasl-creds"
      }
    }
  ]
}

The backup is now running, and will continue to run until you delete it, or pause it.