files/arthur-docs-markdown/platform-management/ongoing-maintenance/backup.md.txt

# Arthur Platform Backup and Restore

## Contents

* Warnings
* Overview
    * Overview - clickhouse-backup
    * Overview - Velero
    * Overview - Arthur (Argo)Workflows
    * Overview - S3
* Pre-requisites
* Installing - Velero
    * Setup ServiceAccount with IAM roles for Backup S3 Bucket
    * Setup CRDs + Cluster-level permissions + Backup Infrastructure
    * Confirm Velero is installed and configured correctly
    * Configure the Backup Storage Destination to Point to S3
* Configuring - clickhouse-backup remote storage
* Backing up
* Restoring
* Appendix
    * Running the Velero CLI
    * Working with Velero - Backup
    * Working with Velero - Restore
    * Backup Architecture

## WARNINGS

### PLEASE READ - FOLLOW THESE INSTRUCTIONS EXACTLY

These instructions have been tested as written. If you find they do not work for your use-case, please reach out to Arthur Support before modifying them. We cannot guarantee reliable operation if these instructions are not followed exactly as written.

### PLEASE READ - TAKE CARE WHEN RESTORING INTO A NEW CLUSTER

When restoring into a new cluster, you must ensure that the new cluster is unable to communicate with any services or data store in the old cluster.

If you took a backup on cluster "Apple", and performed a restore into cluster "Banana", cluster "Banana" must point to its own RDS Instance, ClickHouse Database, and Kafka Store (note: it is ok if clusters share an S3 bucket, but not ideal).

To ensure this, you must re-configure via the Admin Interface when restoring into a new cluster. Failure to do this **WILL CAUSE DATA CORRUPTION** on both clusters that is unrecoverable.

### PLEASE READ - ENSURE CONSISTENCY WITH BACKUPS

If you are either manually taking a backup, or scheduling a backup, you **MUST** take a backup of of the full platform. You **CANNOT** use a ClickHouse snapshot taken at midnight with a RDS snapshot taken at 0400 AM (or any other time). All backup operations must be performed at the same time, and when restoring, the data you are using must all belong to the same backup operation. This is to ensure data consistency across the different data stores. **IGNORING THIS WILL CAUSE DATA CORRUPTION**.

## Overview

The overall backup and restore process for the Arthur Platform is as follows:

* Backup a cluster
	* Take a backup of ClickHouse Data
	* Take a backup of Kubernetes Deployment State and Persistent Volumes
		* Enrichments infrastructure
			* Model Servers
			* Data Pipeline Services
			* Enrichment / Delete Enrichment Workflows
		* Kafka Deployment State and EBS Volumes (using EBS Snapshots)
	* Take a backup of RDS Postgres
* Restore the cluster
	* Restore RDS Postgres
	* Update configuration and install the platform
	* Restore ClickHouse Data
	* Restore the Kafka Deployment State and Persistent Volumes
	* Restore Enrichments infrastructure
	* Restore Workflows
* Smoke Tests and Validation

### Overview - clickhouse-backup

The Arthur Platform stores inference data, data built from the enrichments pipeline, reference and ground truth data in ClickHouse. ClickHouse is an open-source OLAP Database which enables SQL-like query execution, replication, sharding and many additional features.

To backup ClickHouse, the Arthur Platform uses a tool called [clickhouse-backup](https://github.com/AlexAkulov/clickhouse-backup). clickhouse-backup is a sidecar-container included on the ClickHouse pods and is responsible for taking backups, performing restores, and coordinating with remote storage (in this case S3) to store and retrieve backups. clickhouse-backup uses built-in functionality of ClickHouse to take backups and perform restores.


### Overview - Velero

The Arthur Platform uses [Velero](https://velero.io/) as its Backup and Restore tool. Velero is an industry-standard, battle-tested tool for backing up Kubernetes Resources as well as Persistent Volumes.

Arthur uses velero to backup most namespaced kubernetes resources, as well as the EBS Volume Snapshot backups for each PersistentVolumes claimed by the StatefulSets (eg: via PVCs).

Backup data (not including EBS Volume Snapshots) is stored in an S3 bucket which is accessible via a ServiceAccount that is provisioned for the Backup and Restore agent. Backup and Restores are managed by Velero using Kubernetes Custom Resource Definitions (CRDs), which are consumed by the Velero Backup Controller and Restic Agents.

Velero has a feature which also allows backups to be scheduled, using a cron-like configuration. It also provides `ServiceMonitors` which expose metrics via Prometheus, so that operators can monitor backup and restore status and set up alerts for when backups or restores fail.

### Overview - Arthur (Argo)Workflows

The Arthur Platform uses [Argo Workflows](https://argoproj.github.io/argo-workflows/) as a workflow orchestration engine for running certain jobs. Argo installs a handful of Custom Resource Definitions (CRDs) which enable the Argo Workflow services to schedule, execute and update these jobs.

Workflows are dynamically managed, meaning that their definitions are not stored in the Arthur installer script. The Backup and Restore operation accounts for this by treating restoration of Workflows on a case-by-case basis, as follows:

* Enrichments and Delete Enrichments workflows
	* These workflows are created to create and tear-down infrastructure necessary for processing enrichments data (eg: kafka topics, pods which manage the data pipeline for enrichments, etc.)
	* These workflows are idempotent and safe to recover
	* Therefore, these workflows are backed up and restored just like any other Kubernetes Resource during the backup stage
* Batch workflows
	* These workflows are created to manage batch jobs, which are used by clients when uploading large data files to models (inferences and/or ground truths).
	* These workflows are sometimes safe to recover
	* Therefore, these workflows are restored selectively based on what state they were in when the backup was taken
		* Workflows for which Arthur received all of the data from the client are resumed by manually re-submitting them (this is done via an Administrative HTTP endpoint that needs to manually be called)
		* Workflows for which Arthur did not receive all the data from the client will need to be re-submitted. Operators restoring the cluster will need to reach out to affected clients to communicate that their batch workflows should be re-submitted.
* Reference and Cron Workflows
	* Reference Workflows are created for monitoring the upload of reference datasets to S3
		* Reference datasets that were in-flight during a backup will need to be re-uploaded with the SDK.
	* Cron Workflows are scheduled workflows which perform some regular processing (eg: triggering alerts for non-batch inferences)
		* Cron Workflows are meant to be run on a regular schedule. It is safe to wait for the next workflow to be triggered, and therefore, these workflows are not backed up nor restored.

### Overview - S3

The Arthur Platform uses AWS S3 as object storage for storing inference data, reference data, as well as data and trained models for the enrichments pipeline.

Arthur recommends ensuring that the AWS S3 bucket used for this storage is configured with Live Cross-Region Replication so that objects are available in the event of an AWS region outage.

The Arthur Backup solution does not manage consistency with the S3 bucket and other backup data.
The data in S3 is only used in conjuction with data that is stored in Postgres (eg: model definitions), so it's ok if there's data in S3 that isn't represented in Postgres.
Therefore, the S3 bucket for a cluster will always reflect the most up-to-date state, regardless of when a backup was taken.

To read more about S3 Bucket Replication, check out the AWS Documentation:
* https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication.html

## Pre-requisites


The following items must be configured specifically in this capacity in order to use Arthur's Backup and Restore capabilities:

1. Arthur must be configured using external object storage, specifically, S3
2. The access to external storage must be configured using IRSA Annotations
3. In order to use IRSA Annotations, the cluster must be deployed using Amazon EKS

If the following are not true/possible for your deployment, please reach out to Arthur Support so we can discuss.



## Installing - Velero

The only component that needs to be installed separately from Arthur to perform backup and restores is Velero. Below, instructions are provided for setting up Velero to store backups in S3 using Secret and Access keys.

The general overview of the installation is as follows:

1. Create the Velero configuration
	1. Create the policy for accessing the S3 Bucket and taking EBS Snapshots and attach to an IAM User
	2. Generate the Secret and Access keys for the IAM User
	3. Create a Velero-specific credentials file
4. Install Velero
5. Confirm Velero is installed and configured correctly
6. Configure the Backup Storage Destination to Point to S3

### Create the Velero Configuration

The instructions here are taken from the Velero AWS Plugin Documentation, which can be found in Option 1 here:
https://github.com/vmware-tanzu/velero-plugin-for-aws#setup

1. Create the IAM user:

    ```bash
    aws iam create-user --user-name velero
    ```

    If you'll be using Velero to backup multiple clusters with multiple S3 buckets, it may be desirable to create a unique username per cluster rather than the default `velero`.

2. Attach policies to give `velero` the necessary permissions:

    ```
    cat > velero-policy.json <<EOF
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": [
                    "ec2:DescribeVolumes",
                    "ec2:DescribeSnapshots",
                    "ec2:CreateTags",
                    "ec2:CreateVolume",
                    "ec2:CreateSnapshot",
                    "ec2:DeleteSnapshot"
                ],
                "Resource": "*"
            },
            {
                "Effect": "Allow",
                "Action": [
                    "s3:GetObject",
                    "s3:DeleteObject",
                    "s3:PutObject",
                    "s3:AbortMultipartUpload",
                    "s3:ListMultipartUploadParts"
                ],
                "Resource": [
                    "arn:aws:s3:::${BUCKET}/*"
                ]
            },
            {
                "Effect": "Allow",
                "Action": [
                    "s3:ListBucket"
                ],
                "Resource": [
                    "arn:aws:s3:::${BUCKET}"
                ]
            }
        ]
    }
    EOF
    ```
    ```bash
    aws iam put-user-policy \
      --user-name velero \
      --policy-name velero \
      --policy-document file://velero-policy.json
    ```

3. Create an access key for the user:

    ```bash
    aws iam create-access-key --user-name velero
    ```

    The result should look like:

    ```
    {
      "AccessKey": {
            "UserName": "velero",
            "Status": "Active",
            "CreateDate": "2017-07-31T22:24:41.576Z",
            "SecretAccessKey": <AWS_SECRET_ACCESS_KEY>,
            "AccessKeyId": <AWS_ACCESS_KEY_ID>
      }
    }
    ```

4. Create a Velero-specific credentials file (`credentials-velero`) in your local directory:

    ```bash
    [default]
    aws_access_key_id=<AWS_ACCESS_KEY_ID>
    aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>
    ```

    where the access key id and secret are the values returned from the `create-access-key` request.

### Install Velero

To install Velero, first install the Velero CLI. Instructions for how to do this can be found on the Velero Documentation site: https://velero.io/docs/v1.10/basic-install/#install-the-cli

Once the Velero CLI is installed, you can use this to install Velero on the cluster. Please ensure that your kubeconfig is pointing to the cluster, as this is what it used by the Velero CLI to communicate with the cluster.

To install Velero, use the following command, ensuring that the path to the credentials file you generated in the last step is provided correctly:

```
velero install \
    --provider aws \
    --plugins velero/velero-plugin-for-aws:v1.6.0 \
    --bucket $BUCKET \
    --backup-location-config region=$REGION \
    --snapshot-location-config region=$REGION \
    --secret-file ./credentials-velero
```

### Confirm Velero is installed and configured correctly

To confirm that Velero is installed and configured correctly:

1. Open the Kots Admin Interface and navigate to the "Snapshots" tab
2. Click the "Check for Velero" button (see the screenshot below)

![Check for Velero](/_static/images/velero-configuration-check.png)

### Configure the Backup Storage Destination to Point to S3

The final step in configuring Velero is to configure the Backup Storage Destination to point to the S3 Bucket where the backups will be stored.

To do this, add a new "Backup Storage Destination" in the Admin Interface and fill in the details for the S3 Bucket and for the Secret and Access keys.

## Configuring - clickhouse-backup remote storage

Before continuing, please ensure that your cluster is setup and configured as described in the "Pre-requisites" section above.

Configuring clickhouse-backup to store backups in remote storage (eg: S3) can be done in the Kots Admin Interface.

If you've performed the configuration steps as mentioned in the "Pre-requisites" section, you should see the "Enable Olap Database Backup Capabilites" option in the "Olap Database" section (see screenshot)

![Configure Olap Backups](/_static/images/clickhouse-backup.png)

Ensure that:

1. The configuration that points to the bucket is correct
    * The Bucket Name
    * The Bucket Region
2. The ServiceAccount is the same ServiceAccount that you've configured with the IRSA Annotation (if you are not sure, enter the default value)
3. The IAM Role that you are using for the IRSA Annotation has the appropriate permissions to read/write/list from the S3 bucket
    * Note - you can use the same permissions that are described in the Velero Section, just be sure to update the bucket ARN that the permissions apply to
4. The S3 Path is where you want to be storing backups

Once you've configured clickhouse-backup, you can validate that the configuration is correct by trying to take a backup. Refer to the "Take a backup of ClickHouse Data" section below on how to do this.

## Backing up

Note that you can use the following script to run all of these steps together. Read the `Appendix: Running the Velero CLI` for instructions on how to run velero.

```bash
#!/bin/bash

set -euo pipefail
IFS=$'\n\t'

# You need to configure this by getting the name of the storage location
# using the velero CLI
# eg: `velero backup-location get`
storage_location="Put your storage location here"

backup_date=$(DATE +%Y-%m-%d-%H-%M-%S);
name=arthur-backup-$backup_date
echo "Creating a new backup with name $name"

echo "Taking a backup of CH data"
kubectl create job $name-clickhouse-backup \
    --from=cronjob/clickhouse-backup-cronjob
ch_backup_jobname=$(kubectl get jobs -o name | grep "$name-clickhouse-backup")
kubectl wait $ch_backup_jobname \
    --for=condition=complete \
    --timeout=30m

echo "Taking a backup of the enrichments infrastructure"
velero backup create $name-enrichments \
    --namespace=arthurai \
    --include-namespaces=arthurai \
    --selector='component in (kafka-mover-init-connector, model_server)' \
    --include-resources=deployments,services \
    --exclude-resources=clusterrolebindings.rbac.authorization.k8s.io,clusterroles.rbac.authorization.k8s.io,controllerrevisions.apps,endpointslices.discovery.k8s.io,customresourcedefinitions.apiextensions.k8s.io,secrets,configmaps \
    --storage-location=$storage_location \
    --wait

echo "Taking a backup of workflows"
velero backup create $name-workflows \
    --namespace=arthurai \
    --include-namespaces=arthurai \
    --include-resources=workflows \
    --exclude-resources=clusterrolebindings.rbac.authorization.k8s.io,clusterroles.rbac.authorization.k8s.io,controllerrevisions.apps,endpointslices.discovery.k8s.io,customresourcedefinitions.apiextensions.k8s.io,secrets,configmaps \
    --storage-location=$storage_location \
    --wait

echo "Taking a backup of Kafka/Kafka-ZK StatefulSets, their EBS Volumes, and related components"
velero backup create $name-stateful-sets \
    --namespace=arthurai \
    --include-namespaces=arthurai \
    --selector='app in (cp-zookeeper,cp-kafka)' \
    --exclude-resources=clusterrolebindings.rbac.authorization.k8s.io,clusterroles.rbac.authorization.k8s.io,controllerrevisions.apps,endpointslices.discovery.k8s.io,customresourcedefinitions.apiextensions.k8s.io \
    --storage-location=$storage_location \
    --wait

echo "Taking a backup of the RDS database"
aws rds create-db-cluster-snapshot \
    --db-cluster-snapshot-identifier $name-snapshot \
    --db-cluster-identifier RDS_DB_NAME \
    --profile AWS_PROFILE_NAME \
    --region AWS_REGION
```

### Take a backup of ClickHouse Data

By default, the Arthur Platform ships with a Kubernetes CronJob which takes a ClickHouse Backup each day at midnight.

Please see the warning `PLEASE READ - ENSURE DATA CONSISTENCY` at the top if attempting to take a manual backup.

To take a manual backup of ClickHouse Data, you can run the following commands:

```bash
$ kubectl get cronjobs -n arthurai
NAME                              SCHEDULE    SUSPEND   ACTIVE   LAST SCHEDULE   AGE
arthurai-cron-workflow-cron-job   1 0 * * *   False     0        14h             2d18h
arthurai-model-health-cron-job    5 * * * *   False     0        20m             2d18h
clickhouse-backup-cronjob         0 0 * * *   False     0        14h             2d18h
$ kubectl create job clickhouse-backup --from=cronjob/clickhouse-backup-cronjob -n arthurai
job.batch/clickhouse-backup created
$ kubectl get jobs -n arthurai
NAME                                       COMPLETIONS   DURATION   AGE
clickhouse-backup-cronjob-27735840         1/1           8m35s      14m
```

### Take a backup of Enrichments infrastructure and Enrichment / Delete Enrichment Workflows

The Arthur Platform uses Velero to take a backup of the Enrichments Infrastructure, as well as the Enrichments workflows. The backup is orchestrated manually and requires running a command.

The Enrichments infrastructure and Enrichment Workflows are orchestrated as separate backups and will require running 2 separate commands.

To take a manual backup of the Enrichments infrastructure, run the following commands:

```bash
$ backup_date=$(DATE +%Y-%m-%d-%H-%M-%S);
$ name=arthur-backup-$backup_date
$ velero backup create $name-enrichments \
    --namespace=arthurai \
    --include-namespaces=arthurai \
    --selector='component in (kafka-mover-init-connector, model_server)' \
    --include-resources=deployments,services \
    --exclude-resources=clusterrolebindings.rbac.authorization.k8s.io,clusterroles.rbac.authorization.k8s.io,controllerrevisions.apps,endpointslices.discovery.k8s.io,customresourcedefinitions.apiextensions.k8s.io,secrets,configmaps \
    --storage-location=$storage_location \
    --wait
```

To take a manual backup of the Enrichments Workflows, run the following commands:

```bash
$ backup_date=$(DATE +%Y-%m-%d-%H-%M-%S);
$ name=arthur-backup-$backup_date
velero backup create $name-workflows \
    --namespace=arthurai \
    --include-namespaces=arthurai \
    --include-resources=workflows \
    --exclude-resources=clusterrolebindings.rbac.authorization.k8s.io,clusterroles.rbac.authorization.k8s.io,controllerrevisions.apps,endpointslices.discovery.k8s.io,customresourcedefinitions.apiextensions.k8s.io,secrets,configmaps \
    --storage-location=$storage_location \
    --wait
```

### Take a backup of Kafka Deployment State and EBS Volumes (using EBS Snapshots)

The Arthur Platform uses Velero to take a backup of the Kafka Deployment State and EBS Volumes. The backup is orchestrated manually and requires running a command.

To take a manual backup of Kafka, run the following commands:

```bash
$ backup_date=$(DATE +%Y-%m-%d-%H-%M-%S);
$ name=arthur-backup-$backup_date
$ velero backup create $name-zookeeper \
    --namespace=arthurai \
    --include-namespaces=arthurai \
    --selector='app in (cp-zookeeper,cp-kafka)' \
    --exclude-resources=clusterrolebindings.rbac.authorization.k8s.io,clusterroles.rbac.authorization.k8s.io,controllerrevisions.apps,endpointslices.discovery.k8s.io,customresourcedefinitions.apiextensions.k8s.io \
    --storage-location=$storage_location \
    --wait
```

If you are unable to run the Velero CLI locally, you can execute it from the Velero container. See the section **Executing on the Velero Backup Controller Pod** for more details.


### Take a backup of RDS Postgres

The script above provides the command line instructions for taking a backup of a RDS Database (also copied here). Please ensure that the values for the `db-cluster-identifier`, `profile` and `region` are filled in correctly.

Note that the command is only compatible for a multi-region RDS Database. If you are using a single-region RDS Database, the command to use is `aws rds create-db-snapshot`.

```bash
aws rds create-db-cluster-snapshot \
    --db-cluster-snapshot-identifier $name-snapshot \
    --db-cluster-identifier RDS_DB_NAME \
    --profile AWS_PROFILE_NAME \
    --region AWS_REGION
```

For more information, please refer to the AWS Documentation:
* Multi-region RDS instance:
    * https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_CreateMultiAZDBClusterSnapshot.html
    * https://awscli.amazonaws.com/v2/documentation/api/latest/reference/rds/create-db-cluster-snapshot.html
* Single-region RDS instance:
    * https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_CreateSnapshot.html
    * https://awscli.amazonaws.com/v2/documentation/api/latest/reference/rds/create-db-snapshot.html

## Restoring

### Restore the RDS Postgres

To restore RDS Postgres, use the following resource:
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_RestoreFromMultiAZDBClusterSnapshot.Restoring.html

Please ensure that you correctly match the following configuration from the DB from which the snapshot was taken
* The connection port
* The VPC and Security Group IDs
* DB Subnet Group
* DB Instance Type
* Any other configuration which might be overridden

This operation might take a while and the DB must show as Available before installing the platform.

### Update configuration and install the platform

**NOTE - Before completing this action, please check the following:**

1. The Restored RDS DB Instance is showing as "Available"
2. All pods which were restored are showing as "Running"

**NOTE - IMPORTANT -- Update configuration to point to the new DB Instance**

It is **CRITICALLY IMPORTANT** to update your configuration to point to the newly restored DB Instance. Failure to complete this step **WILL CAUSE DATA CORRUPTION**.

Please update the configuration in the "Meta Database" section of the Admin Interface to point to the newly restored DB instance.

If your new cluster is routed via a different ingress URL, please also update this in the "Network" section of the configuration.

Wait for the platform to come back online before proceeding to the next steps. All Deployments and StatefulSets should be completely stood up (eg: all Pods should be ready and "Running") and all Jobs should be "Completed".


### Restore ClickHouse Data

The Arthur Platform ships with a Kubernetes CronJob that executes a ClickHouse restore that is scheduled to never run (eg: it's scheduled to run on February 31st, which is an invalid date).

To run a ClickHouse data restore, execute the following commands:

First, we need to get the name of the clickhouse-backup that coincides with the kafka/enrichments/workflow backups that you are restoring:

```bash
$ kubectl get pods | grep olap
chi-olap-installation-arthur-0-0-0                           2/2     Running     0          6d23h
chi-olap-installation-arthur-0-1-0                           2/2     Running     0          6d23h
chi-olap-installation-arthur-0-2-0                           2/2     Running     0          6d23h
olap-installation-zookeeper-0                                3/3     Running     0          6d23h
olap-installation-zookeeper-1                                3/3     Running     0          6d22h
olap-installation-zookeeper-2                                3/3     Running     0          6d23h
olap-operator-8c867ddff-kc22x                                2/2     Running     0          6d23h
$ kubectl exec chi-olap-installation-arthur-0-0-0 -c backup -- clickhouse-backup list
2022/10/04 16:03:20.255378  info SELECT value FROM `system`.`build_options` where name='VERSION_INTEGER'
2022/10/04 16:03:20.257582  info SELECT * FROM system.disks;
2022/10/04 16:03:20.267149  info SELECT count() AS is_macros_exists FROM system.tables WHERE database='system' AND name='macros'
2022/10/04 16:03:20.271359  info SELECT * FROM system.macros
chi-olap-installation-arthur-0-0-arthur-clickhouse-backup-2022-09-26-19-50-43   3.64MiB   02/10/2022 00:00:16   remote      tar, regular
chi-olap-installation-arthur-0-1-arthur-clickhouse-backup-2022-09-26-19-50-43   3.64MiB   02/10/2022 00:00:16   remote      tar, regular
chi-olap-installation-arthur-0-2-arthur-clickhouse-backup-2022-09-26-19-50-43   3.64MiB   02/10/2022 00:00:16   remote      tar, regular
```

Note that from the above backup, the backup names are in the following format:
`$CLICKHOUSE_NODE_NAME-$ARTHUR_BACKUP_NAME`

So for example, `chi-olap-installation-arthur-0-0-arthur-clickhouse-backup-2022-09-26-19-50-43` can be parsed into:
* clickhouse node name: `chi-olap-installation-arthur-0-0`
* arthur backup name: `arthur-clickhouse-backup-2022-09-26-19-50-43`

Now we need to create the restoration job, and configure it to use the Arthur Backup Name from above:

```bash
$ kubectl create job --from=cronjob/clickhouse-restore-cronjob -o yaml clickhouse-restore --dry-run=client --save-config > clickhouse-restore.yaml

# update the value of the `BACKUP_NAME` environment variable in the `clickhouse-restore.yaml` file
# eg:
#        - name: BACKUP_NAME
#          value: "arthur-clickhouse-backup-2022-09-26-19-50-43"

$ kubectl apply -f clickhouse-restore.yaml
job.batch/clickhouse-restore created
```

### Restore the Kafka Deployment State and Persistent Volumes

The Arthur Platform restores Kafka Deployment State and PVs using Velero.

To execute a restore, run the following commands:

```bash
$ velero backup get -n arthurai | grep kafka
NAME                                                    STATUS      ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
arthur-backup-2022-09-23t11.23.25-04.00-kafka           Completed   0        0          2022-09-23 11:24:37 -0400 EDT   27d       default            app in (cp-kafka,cp-zookeeper)
$ velero restore create \
    --from-backup "arthur-backup-2022-09-23t11.23.25-04.00-kafka" \
    --namespace arthurai \
    --restore-volumes=true \
    --existing-resource-policy=update
```

Velero will update the Pod Specs and point to the PVs using the EBS Volume Snapshots and restore the kubernetes resources associated with Kafka.

### Restore Enrichments infrastructure

The Arthur Platform uses Velero to restore the Enrichments infrastructure.

To restore, run the following commands:

```bash
$ velero backup get -n arthurai | grep enrichments
NAME                                                    STATUS      ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
arthur-backup-2022-09-23t11.23.25-04.00-enrichments     Completed   0        0          2022-09-23 11:24:33 -0400 EDT   27d       default            component in (kafka-mover-init-connector,model_server)
$ velero restore create \
    --from-backup "arthur-backup-2022-09-23t11.23.25-04.00-enrichments" \
    --namespace arthurai
```

### Restore Workflows

Restoring workflows is a 2-step process:

1. Restore the workflows from the Velero backup
2. Restore Batch Workflows which are recoverable using an Arthur Admin Endpoint

To restore workflows using the Velero backup, run the following commands:

```bash
$ velero backup get -n arthurai | grep workflows
NAME                                                    STATUS      ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
arthur-backup-2022-09-23t11.23.25-04.00-workflows       Completed   0        0          2022-09-23 11:24:35 -0400 EDT   27d       default            <none>
$ velero restore create \
    --from-backup "arthur-backup-2022-09-23t11.23.25-04.00-workflows" \
    --namespace arthurai
```

To restore Batch Workflows, run the following comamnds.

In one terminal window, port-forward to the dataset-service:

```bash
$ kubectl get pods | grep dataset-service
arthurai-dataset-service-69dd979d8c-72t7j                         1/1     Running            0          2d17h
$ kubectl port-forward -n arthurai arthurai-dataset-service-69dd979d8c-72t7j 7899
```

Then, in another terminal window, run the following commands:

```bash
$ curl -k -XPOST https://localhost:7899/api/v1/workflows/batch/recover
{"message":"success"}
```

### Smoke Tests and Validation

The restore should now be complete. All data should be restored and in a consistent state as it was from when the backup was taken. Any data sent during or after the backup will need to be re-sent. Perform any validation/smoke-tests to ensure that the platform is operating.

## Appendix

### Running the Velero CLI

Velero provides a Command-Line Interface (CLI) for taking backups and performing restores. The command line interface can be installed locally, or it can be invoked by `kubectl exec` on the Velero Backup Controller pod.

#### Local Installation

Read through the Velero Documentation for how to install velero on your platform:
https://velero.io/docs/v1.9/basic-install/#install-the-cli

Velero uses your KUBECONFIG file to connect to the cluster.

```
$ velero --help
Velero is a tool for managing disaster recovery, specifically for Kubernetes
cluster resources. It provides a simple, configurable, and operationally robust
way to back up your application state and associated data.

If you're familiar with kubectl, Velero supports a similar model, allowing you to
execute commands such as 'velero get backup' and 'velero create schedule'. The same
operations can also be performed as 'velero backup get' and 'velero schedule create'.

Usage:
  velero [command]

Available Commands:
  backup            Work with backups
  backup-location   Work with backup storage locations
  bug               Report a Velero bug
  client            Velero client related commands
  completion        Generate completion script
  create            Create velero resources
  debug             Generate debug bundle
  delete            Delete velero resources
  describe          Describe velero resources
  get               Get velero resources
  help              Help about any command
  install           Install Velero
  plugin            Work with plugins
  restic            Work with restic
  restore           Work with restores
  schedule          Work with schedules
  snapshot-location Work with snapshot locations
  uninstall         Uninstall Velero
  version           Print the velero version and associated image

Flags:
      --add_dir_header                   If true, adds the file directory to the header
      --alsologtostderr                  log to standard error as well as files
      --colorized optionalBool           Show colored output in TTY. Overrides 'colorized' value from $HOME/.config/velero/config.json if present. Enabled by default
      --features stringArray             Comma-separated list of features to enable for this Velero process. Combines with values from $HOME/.config/velero/config.json if present
  -h, --help                             help for velero
      --kubeconfig string                Path to the kubeconfig file to use to talk to the Kubernetes apiserver. If unset, try the environment variable KUBECONFIG, as well as in-cluster configuration
      --kubecontext string               The context to use to talk to the Kubernetes apiserver. If unset defaults to whatever your current-context is (kubectl config current-context)
      --log_backtrace_at traceLocation   when logging hits line file:N, emit a stack trace (default :0)
      --log_dir string                   If non-empty, write log files in this directory
      --log_file string                  If non-empty, use this log file
      --log_file_max_size uint           Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
      --logtostderr                      log to standard error instead of files (default true)
  -n, --namespace string                 The namespace in which Velero should operate (default "velero")
      --skip_headers                     If true, avoid header prefixes in the log messages
      --skip_log_headers                 If true, avoid headers when opening log files
      --stderrthreshold severity         logs at or above this threshold go to stderr (default 2)
  -v, --v Level                          number for the log level verbosity
      --vmodule moduleSpec               comma-separated list of pattern=N settings for file-filtered logging

Use "velero [command] --help" for more information about a command.
```

#### Executing on the Velero Backup Controller Pod

```
$ kubectl exec velero-699dc869d4-r24bh -c velero -- /velero
Velero is a tool for managing disaster recovery, specifically for Kubernetes
cluster resources. It provides a simple, configurable, and operationally robust
way to back up your application state and associated data.

If you're familiar with kubectl, Velero supports a similar model, allowing you to
execute commands such as 'velero get backup' and 'velero create schedule'. The same
operations can also be performed as 'velero backup get' and 'velero schedule create'.

Usage:
  velero [command]

Available Commands:
  backup            Work with backups
  backup-location   Work with backup storage locations
  bug               Report a Velero bug
  client            Velero client related commands
  completion        Generate completion script
  create            Create velero resources
  debug             Generate debug bundle
  delete            Delete velero resources
  describe          Describe velero resources
  get               Get velero resources
  help              Help about any command
  install           Install Velero
  plugin            Work with plugins
  restic            Work with restic
  restore           Work with restores
  schedule          Work with schedules
  snapshot-location Work with snapshot locations
  uninstall         Uninstall Velero
  version           Print the velero version and associated image

Flags:
      --add_dir_header                   If true, adds the file directory to the header
      --alsologtostderr                  log to standard error as well as files
      --colorized optionalBool           Show colored output in TTY. Overrides 'colorized' value from $HOME/.config/velero/config.json if present. Enabled by default
      --features stringArray             Comma-separated list of features to enable for this Velero process. Combines with values from $HOME/.config/velero/config.json if present
  -h, --help                             help for velero
      --kubeconfig string                Path to the kubeconfig file to use to talk to the Kubernetes apiserver. If unset, try the environment variable KUBECONFIG, as well as in-cluster configuration
      --kubecontext string               The context to use to talk to the Kubernetes apiserver. If unset defaults to whatever your current-context is (kubectl config current-context)
      --log_backtrace_at traceLocation   when logging hits line file:N, emit a stack trace (default :0)
      --log_dir string                   If non-empty, write log files in this directory
      --log_file string                  If non-empty, use this log file
      --log_file_max_size uint           Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
      --logtostderr                      log to standard error instead of files (default true)
  -n, --namespace string                 The namespace in which Velero should operate (default "arthurai")
      --skip_headers                     If true, avoid header prefixes in the log messages
      --skip_log_headers                 If true, avoid headers when opening log files
      --stderrthreshold severity         logs at or above this threshold go to stderr (default 2)
  -v, --v Level                          number for the log level verbosity
      --vmodule moduleSpec               comma-separated list of pattern=N settings for file-filtered logging

Use "velero [command] --help" for more information about a command.
```

### Working with Velero - Backup

To take a backup of Arthur, you would invoke the CLI as follows.

```bash
$ velero backup create arthur-backup-$(date -Iseconds | tr ":" "." | tr "T" "t") \
    --namespace=arthurai \
    --include-namespaces=arthurai
```

The command above will create a Velero Backup Resource named `arthur-backup-<timestamp>`, which you can check using the Velero CLI:

```bash
$ velero backup get -n arthurai
NAME                                      STATUS       ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
arthur-backup-2022-08-25t09.59.35-04.00   Completed    0        0          2022-08-25 09:59:35 -0400 EDT   28d       default            <none>
arthur-backup-2022-08-26t10.37.52-04.00   InProgress   0        0          2022-08-26 10:37:52 -0400 EDT   29d       default            <none>
```

For debugging a backup, you can access the backup's logs using the Velero CLI:

```bash
$ velero backup logs arthur-backup-2022-08-25t09.59.35-04.00 -n arthurai | head
time="2022-08-25T13:59:35Z" level=info msg="Setting up backup temp file" backup=arthurai/arthur-backup-2022-08-25t09.59.35-04.00 logSource="pkg/controller/backup_controller.go:587"
time="2022-08-25T13:59:35Z" level=info msg="Setting up plugin manager" backup=arthurai/arthur-backup-2022-08-25t09.59.35-04.00 logSource="pkg/controller/backup_controller.go:594"
time="2022-08-25T13:59:35Z" level=info msg="Getting backup item actions" backup=arthurai/arthur-backup-2022-08-25t09.59.35-04.00 logSource="pkg/controller/backup_controller.go:598"
time="2022-08-25T13:59:35Z" level=info msg="Setting up backup store to check for backup existence" backup=arthurai/arthur-backup-2022-08-25t09.59.35-04.00 logSource="pkg/controller/backup_controller.go:608"
time="2022-08-25T13:59:36Z" level=info msg="Writing backup version file" backup=arthurai/arthur-backup-2022-08-25t09.59.35-04.00 logSource="pkg/backup/backup.go:192"
time="2022-08-25T13:59:36Z" level=info msg="Including namespaces: arthurai" backup=arthurai/arthur-backup-2022-08-25t09.59.35-04.00 logSource="pkg/backup/backup.go:198"
time="2022-08-25T13:59:36Z" level=info msg="Excluding namespaces: <none>" backup=arthurai/arthur-backup-2022-08-25t09.59.35-04.00 logSource="pkg/backup/backup.go:199"
time="2022-08-25T13:59:36Z" level=info msg="Including resources: *" backup=arthurai/arthur-backup-2022-08-25t09.59.35-04.00 logSource="pkg/backup/backup.go:202"
time="2022-08-25T13:59:36Z" level=info msg="Excluding resources: <none>" backup=arthurai/arthur-backup-2022-08-25t09.59.35-04.00 logSource="pkg/backup/backup.go:203"
time="2022-08-25T13:59:36Z" level=info msg="Backing up all pod volumes using Restic: false" backup=arthurai/arthur-backup-2022-08-25t09.59.35-04.00 logSource="pkg/backup/backup.go:204"

<truncated>
```

Finally, you can get an overview of the backup using the Velero CLI:

```bash
$ velero backup describe arthur-backup-2022-08-25t09.59.35-04.00 -n arthurai
Name:         arthur-backup-2022-08-25t09.59.35-04.00
Namespace:    arthurai
Labels:       velero.io/storage-location=default
Annotations:  velero.io/source-cluster-k8s-gitversion=v1.21.14-eks-18ef993
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=21+

Phase:  Completed

Errors:    0
Warnings:  0

Namespaces:
  Included:  arthurai
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Storage Location:  default

Velero-Native Snapshot PVs:  auto

TTL:  720h0m0s

Hooks:  <none>

Backup Format Version:  1.1.0
szExpiration:  2022-09-24 09:59:35 -0400 EDT

Total items to be backed up:  654
Items backed up:              654

Velero-Native Snapshots:  3 of 3 snapshots completed successfully (specify --details for more information)

Restic Backups (specify --details for more information):
  Completed:  23
```


### Working with Velero - Restore

Similar to backup, restore happens using the Velero Cli. A restore takes a Backup object and then executes the restore procedure.

You can execute a restore with the following Velero CLI command:

```bash
$ velero restore create \
    --from-backup arthur-backup-2022-08-25t09.59.35-04.00 \
    --namespace arthurai \
    --restore-volumes=true
```

Just like with the Backup, Velero will create a Restore Velero Resource, which you can inspect with the Velero CLI:

```bash
$ velero restore get -n arthurai
NAME                                                     BACKUP                                    STATUS            STARTED                         COMPLETED                       ERRORS   WARNINGS   CREATED                         SELECTOR
arthur-backup-2022-08-26t10.37.52-04.00-20220826110520   arthur-backup-2022-08-26t10.37.52-04.00   Completed         2022-08-26 11:06:16 -0400 EDT   2022-08-26 11:08:00 -0400 EDT   0        35         2022-08-26 11:05:21 -0400 EDT   <none>
```