FlashGrid Cluster on AWS. Backup Best Practices – FlashGrid Help Center

This articles covers backup and, where applicable, restore procedures recommended for different types of data including:

Backing up and restoring OS and software volume on the cluster nodes
Restoring a terminated instance
Backing up and restoring database files
Backing up Grid infrastructure configuration files
Backing up and Restoring files on ACFS

Backing up OS (root) and software volume on the cluster nodes

It is strongly recommended to back up the OS and software volumes of all cluster nodes after the initial cluster configuration and before and after applying any changes, such as patch installation or security settings changes.

The OS (or root) volume for Flashgrid instances is generally identified as /dev/sda1

The software volume for Flashgrid instances (/u01 filesystem) is generally identified as /dev/xvdz

OS and software on a cluster node can be backed up by one or more of the following methods:

creating an AMI for the instance
creating snapshots of individual EBS volumes
using AWS Backup to run a one-off or schedule backup of EBS volumes

AMI-based backup is recommended as it allows easier way to recover an instance in case it is terminated because of a failure or a human error.

To create a backup AMI for a cluster node:

Flush system cache using
```
# sync
```
command on the system.
Create AMI for the instance without rebooting the instance.

IMPORTANT! Remove data volumes from the list of volumes that will be backed up to the AMI. Typically only the root volume (/dev/sda1) and the software volume (/dev/xvdz), if present, must be included in the backup AMI. Other volumes must be excluded. Failure to exclude data volumes from the AMI will create inconsistency in ASM disk groups if the data volumes are later restored from the AMI. Data volumes should never be backed up or restored at the volume level. Instead, database/ACFS file backup procedures must be used as described further in this article.

To create a snapshot of individual EBS volumes

Flush system cache using
```
# sync
```
command on the system.
Locate the volume that requires a snapshot via the AWS EC2 Console, and select Actions -> Create Snapshot

AWS Backup for EBS volumes

A one-time (On-demand) or scheduled (Backup plan) backup can be created for the root (/dev/sda1) and software volume (usually /dev/xvdz) for each server in a cluster.

For an On-demand backup, only one EBS volume can be backed up at a time. Choose EBS as the Resource type, and manually select the volume by Volume ID.

For a scheduled backup, either

add the root & software volumes, identified by Volume ID, in Resource assignments, or
tag the EBS volumes for backup using unique tag/value combinations, and use this in Refine selection using tags section on the Assign resources page.

Note: We do not recommend using AWS Backup for EC2 as there is no way to exclude data (ASM) disks. Data volumes should never be backed up or restored at the volume level. Instead, database/ACFS file backup procedures must be used as described further in this article

Restoring OS (root) or software volume of a cluster node

This section outlines how to manually restore a failed or corrupted OS (root) or software volume from a backup using the AWS Management Console (requires node downtime).

Alternatively If you only need to restore the OS (root) volume and the node is in a running state, you can use the method in the next section Restoring OS (root) of a cluster node (instance in a running state)

Before beginning, verify your volume types and typical device mappings:

OS (root) volume (/dev/sda1): The primary EBS volume containing the Operating System
Software volume (typically /dev/xvdz): The EBS volume hosting the Oracle software binaries (contains the /u01 directory)

To restore root volume (/dev/sda1) or software volume (/dev/xvdz):

Identify the Snapshot ID.

Choose one of the following methods depending on your backup source:
1. Restoring from an EBS snapshot
  - Open the AWS EC2 console and navigate to Instances -> Instances
  - Select the affected instance
  - Open the Storage section, and identify the Volume ID for the root volume (device name /dev/sda1) or software volume (device name generally /dev/xvdz), for example vol-06fb3445aa40ac0cc
  - Open Elastic Block Store -> Snapshots
  - In the Search bar, list all snapshots associated with that volume using the search function - the following syntax as an example: "Volume ID = vol-06fb3445aa40ac0cc"
  - Identify the Snapshot ID to restore from, for example snap-01daff6d71977996a
2. Restoring from a Backup AMI Image
  - Locate the backup AMI for the affected instance and identify the Snapshot ID (displayed as Device ID) linked to the target volume, for example snap-0d8e7dec7009ddc43
3. Restoring from AWS Backup (EBS)
  - Go to the AWS Backup console and open the Protected Resources page.
  - Click the Resource ID that matches the volume ID you want to restore (e.g. volume/vol-0729311a92cdaa731)
  - Find the correct Recovery Point ID based on the creation time, and identify the Snapshot ID, for example snap-08bd60eab2aaeecc2
Create the New Volume
- Open the AWS EC2 console and navigate to Elastic Block Store -> Snapshots
- Search for the snapshot identified in Step 1
- Select that snapshot, and choose Actions -> Create volume from Snapshot
- Ensure Volume type is General Purpose SSD (gp3)
- Double-check that the Size, IOPS, Throughput and Availability Zone match your original volume settings
- Click on Create volume and copy the new volume ID.
  Note it may some time for the volume initialization to complete. The new volume can be used immediately, however if it is attached and used before Initialization state reaches 100% this can affect performance when reading from the volume.
Swap the Volumes
1. Stop the affected instance:
  - If the OS is running then stop the instance gracefully using
```
# flashgrid-node poweroff
```
  - If the OS is not running then stop the instance using AWS console or CLI
2. Detach the affected volume from the instance
3. Attach the newly created volume to the instance
  Note: You must use the exact same device name as the original (/dev/sda1 for the root volume, or /dev/xvdz for the software volume).
4. Start the instance

Restoring OS (root) volume of a cluster node (instance in a running state)

Amazon EC2 allows you to replace the OS (root) volume of a running instance. The replacement process runs in the background and the instance automatically reboots with the new OS (root) volume once the restoration action is initialized.

If the OS (root) & software volumes both need to be restored, use the process described in the previous section Restoring OS (root) or software volume of a cluster node

To replace the OS (root) volume on a running cluster node:

Identify either the AMI or volume snapshot that you wish to restore from (which must be from the server that is being restored).
Make sure there are no other nodes that are in offline or re-syncing state. All disk groups must have zero offline disks and Resync = No:
```
# flashgrid-cluster
```
If the node is a database node,

a. Stop all local database instances running on the node.

b. Stop Oracle CRS on the node:
```
# crsctl stop crs
```
Stop FlashGrid Storage Fabric services on the node (required if it is a database or quorum node):
```
# flashgrid-node stop
```
In the Amazon EC2 console, select the instance for which to replace the root volume and choose Actions -> Monitor and troubleshoot -> Replace root volume.
Note: The Replace root volume action is disabled if the selected instance is not in the running state.
For Restore, choose one of the following options (do not choose the Launch State option)
- Snapshot – Restore the replacement root volume to the snapshot that you specify. For Snapshot, select the snapshot to use.
- Image – Restore the replacement root volume using the AMI that you specify. For Image, select the AMI to use.
(Optional) For Volume initialization rate, you can specify the Amazon EBS Provisioned Rate for Volume Initialization (volume initialization rate), in MiB/s, at which the snapshot blocks are to be downloaded from Amazon S3 to the volume. To use the default initialization rate or fast snapshot restore (if it is enabled for the selected snapshot), do not specify a rate.
(Optional) To delete the root volume that you are replacing, select Delete replaced root volume (it is recommended that this not selected. Instead, delete the volume manually at a later time).
Choose Create replacement task.
The instance will be rebooted almost immediately, with a new root volume created and attached from the chosen backup.
Note: Although the new root volume is immediately created and attached, it will take some time to complete initialization. While the initialization is underway, read performance for the root volume may be slower than usual. To monitor progress select the root volume (Device name /dev/sda1) from the instance Storage tab and view the Initialization state. A value below 100% indicates that the volume is still restoring from snapshot.

Restoring an instance that was accidentally terminated

Setting instance termination protection is strongly recommended to prevent accidental termination of a cluster node instance.

To restore an instance that was terminated

Launch a new instance using the backup AMI for the instance that was terminated, OR, if recovering from an AWS Backup (EBS), locate the root volume backup by snapshot id, and create a new AMI from that snapshot

Make sure that correct instance type, VPC, subnet, and security group are selected.
Configure the same Primary IP that was used on the terminated instance.
Specify a placement group corresponding to the cluster
Make sure that only /dev/sda1 and /dev/xvdz (only on database nodes) volumes are configured. Remove any other volumes if they are present in the AMI.

Attach data volumes to the new instance using the same device names (such as xvdba) that were previously used
Log in to the instance and bring the data disks online:
```
$ flashgrid-node online
```

Backingup and restoring database files

Use standard RMAN procedures for backing up and restoring database files. The two recommended options for backup storage destination are:

Amazon S3. Provides maximum flexibility with easy shared access to the backup files.
An EBS volume with a local file system. Provides maximum performance, with up to 500 MB/s of read/write bandwidth on a st1 type of volume.

For information about backing up to S3 see the following documentation from Oracle:

To configure an EBS volume as a backup storage destination

Create an EBS volume in the availability zone where the instance running RMAN is located. st1 volume type is recommended.
Attach the volume to the instance running RMAN. Select a device name in the xvdc to xvdg range - disks in this name range will be treated as local and will not be shared by FlashGrid Storage Fabric.
Format the volume with a local file system (XFS recommended) and create a mount point for it.
Use standard RMAN procedures to configure backup to the local file system.

Note that an EBS volume can be moved only between instances in the same availability zone. However, snapshot of the volume can be used to clone the volume to a different availability zone.

Backing up Grid Infrastructure configuration files

Please follow Backup Best Practices: Grid Infrastructure Configuration

Backing up and restoring files on ACFS

For backing up and restoring files on ACFS use same tools and procedures that you would normally use for file-level backup and restore.