Skip to main content
FlashGrid.io

FlashGrid Cluster on AWS. Deployment Guide

  • Updated
  • Products: FlashGrid Cluster on AWS

rev. 26.05

Table of Contents

1 Overview
    1.1 Key Components
    1.2 High Availability Architecture
    1.3 Infrastructure-as-Code Deployment
2 Prerequisites
    2.1 Required Knowledge
    2.2 Getting access to FlashGrid Cluster AMI from AWS Marketplace
    2.3 Uploading Oracle installation files to S3
    2.4 Preparing the VPC
3 Deploying a Cluster
4 After Deploying a Cluster
    4.1 Verifying Cluster Status
    4.2 OS User Accounts
    4.3 Finalizing Cluster Configuration
    4.4 Enabling Termination Protection
    4.5 Installing an Additional Database Home
    4.6 Use of Anti-virus and Other Third-party software
    4.7 Use of Automatic Configuration Tools
    4.8 Security Hardening
    4.9 Convert RHEL PAYG to BYOS License
5 Monitoring Cluster Health
6 Before Going Live
7 Deleting a Cluster
8 Additional Documentation
9 Technical Support

1 Overview

FlashGrid Cluster is a virtual clustered appliance that enables deployment of Oracle databases with Oracle RAC or Failover HA on Amazon EC2 instances for mission-critical workloads. Oracle RAC nodes can be distributed across AWS Availability Zones to support database uptime of 99.99% or higher, depending on the deployment architecture and availability requirements. Deployment is fully automated using Infrastructure-as-Code and typically completes in 30 to 90 minutes. FlashGrid provides 24/7 technical support covering the complete infrastructure stack used by the cluster. First introduced on AWS in 2017, FlashGrid Cluster is available in all AWS regions, including AWS GovCloud (US), AWS European Sovereign Cloud, and AWS China.

Detailed information about FlashGrid Cluster architecture for Oracle databases on AWS is available on the following pages:

1.1 Key Components

Key components of FlashGrid Cluster on AWS:

  • FlashGrid Storage Fabric software
  • FlashGrid Cloud Area Network software
  • Oracle Database: 26ai, 19c, 12.2.0.1, 12.1.0.2, or 11.2.0.4
  • Oracle Grid Infrastructure: 26ai or 19c
  • Operating System:
    • Oracle Linux: 8 (UEKR7) or 9 (UEKR7)
    • Red Hat Enterprise Linux (RHEL): 8 or 9
  • Amazon EC2 instances:
    • General purpose: M8i, M8a, M8azn, M7i, M7a, M6i, M6in, M6a
    • Memory optimized: R8i, R8a, R7i, R7iz, R7a, R6i, R6in, R6a, R5b, High Memory, X8i, X8aedz, X2idn, X2iedn, X2iezn, Z1d
  • Disks:
    • EBS GP3 volumes (default)
    • EBS SC1 may be used as bulk second-tier storage. For example, for storing Large Object (LOB) tablespaces in a separate disk group.
    • EBS GP2 volumes (AWS Outposts only)

mceclip0.png

FlashGrid Cluster Network Diagram

1.2 High Availability Architecture

By leveraging Oracle RAC active-active database clustering and synchronous data mirroring across nodes and AZs, FlashGrid Cluster enables near-zero (seconds) Recovery Time Objective (RTO) and zero Recovery Point Objective (RPO) in case of a failure of a single node instance, of a single EBS volume, or in case of one AZ failure.

If the cluster has 3+ database nodes, then two simultaneous database node failures can be tolerated without causing loss of database service.

Standard Oracle Client functionality provides mechanisms for application failover from a failed node, including Transparent Application Failover (TAF)

1.3 Infrastructure-as-Code Deployment

FlashGrid Cluster is delivered as an AWS CloudFormation or Terraform template that automates configuration of multiple components required for a database cluster. FlashGrid Launcher is an online tool that simplifies the deployment process by guiding through the cluster configuration parameters and generating CloudFormation or Terraform templates.

2 Prerequisites

2.1 Required Knowledge

Working knowledge of the following AWS services is required for successful deployment of FlashGrid Cluster on AWS: EC2, VPC, EBS, CloudFormation, S3, IAM, Marketplace.

2.2 Getting access to FlashGrid Cluster AMI from AWS Marketplace

To be able to create a cluster your AWS account must have an active subscription to the selected FlashGrid Cluster AMI. Otherwise, deployment will fail when creating EC2 instances. The FlashGrid Cluster AMIs are based on either Oracle Linux or RHEL.

To get access to the FlashGrid Cluster AMI

  1. Open FlashGrid Cluster product page in AWS Marketplace

    Commercial and GovCloud regions:

    European Sovereign Cloud (credentials required to view products):

  2. Click View Purchase Options button
  3. Click Accept Terms button

Software fees charged through AWS Marketplace include FlashGrid Cluster software license and 24x7 Mission-Critical support plan. The fees are charged by the hour per instance and depend on the selected EC2 instance type and size. Pricing information is available on the AWS Marketplace product pages – see the links above.

2.3 Uploading Oracle installation files to S3

During cluster initialization Oracle installation files will be downloaded from an S3 bucket. The list of files that must be placed in the S3 bucket will be shown by the FlashGrid Launcher tool. The same S3 bucket can be used for deploying multiple clusters.

Please refer to this Knowledge Base article for the steps to upload Oracle installation files to S3.

2.4 Preparing the VPC

When creating a new cluster, you have two options:

  • Automatically create a new VPC.

    This option is usually used for test clusters isolated in their own sandbox VPCs. A VPC will be created together with the required subnets, placement group(s), and security groups. By default, the VPC will be created with CIDR 10.100.0.0/16

  • Create the cluster in an existing VPC.

    This option is used for majority of production deployments where other systems (e.g. app servers) share the same VPC as the cluster. You will need to have subnets and security groups ready and provide their IDs in the FlashGrid Launcher tool and in the CloudFormation Manager.

If using an existing VPC then make sure that the following pre-requisites are met before creating a cluster:

  • The VPC may have any CIDR that does not overlap with 192.168.0.0/16, for example 10.100.0.0/16. If you must use VPC with CIDR that overlaps with 192.168.0.0/16 then please request a customized configuration file from FlashGrid support.
  • The VPC has a subnet in each of the availability zones used for the cluster nodes.
  • The VPC has an S3 endpoint configured (required unless public IPs can be enabled for access to S3)
  • If you choose to enable Public IPs on the EC2 instances, then the VPC must have Internet Gateway configured.
  • Create a security group in the VPC with the following ports open:
    • Inbound and Outbound: All traffic between members of the security group (cluster nodes).
    • Inbound: TCP ports 1521, 1522 for SCAN and Local Listener access to the database nodes from app servers and other database clients. These are default port numbers that can be changed in the FlashGrid Launcher tool.
    • Inbound: TCP port 22 for SSH access to the cluster nodes
    • Inbound: TCP port 5901 if you choose to use VNC for creating a database using DBCA in GUI mode with direct connection (vs. SSH tunnel)
    • Inbound access to the ports listed above must be allowed only from those security groups or IP ranges that require such access. Do not configure Anywhere or 0.0.0.0/0 as allowed sources.

3 Deploying a Cluster

The FlashGrid Launcher tool simplifies deployment of Oracle RAC clusters in AWS by automating the following tasks:

  • Creating and configuring EC2 VPC, subnets, security group (optional)
  • Creating EBS volumes and launching EC2 instances for all nodes in the cluster
  • Installing and configuring FlashGrid Cloud Area Network
  • Installing and configuring FlashGrid Storage Fabric
  • Installing and patching Oracle Grid Infrastructure software
  • Configuring Grid Infrastructure cluster
  • Installing and patching Oracle Database software
  • Creating ASM disk groups

Note: The deployment doesn’t require root privileges to follow the principle of least privilege.

To create a cluster with CloudFormation

  1. Log in to AWS Management Console with a user account that has the following privileges:
    • AWSCloudFormationFullAccess
    • AmazonEC2FullAccess
    • AmazonVPCFullAccess (required only if creating a new VPC)
  2. Open FlashGrid Launcher tool:
  3. Configure parameters of the cluster
  4. Click Validate Configuration button
  5. If verification passes then click Launch Cluster button, which will take you to AWS CloudFormation Manager
  6. Click Next
  7. Select your SSH key
  8. If using an existing VPC, then select subnet(s), and a security group.
  9. Click Next
  10. On the Options page:
    • If you added tags in FlashGrid Launcher, then do not add the same tags in CloudFormation Manager
    • If the cluster is for production use, then expand the Advanced options and enable Termination Protection
  11. Click Next
  12. Click Create
  13. Wait until the status of the stack changes to CREATE_COMPLETE
  14. If creating the stack fails:
    1. Check for the cause of the failure on the Events tab
    2. Correct the cause of the error
    3. Delete the failed stack
    4. Repeat the steps for creating a new stack
  15. Use EC2 Management Console to get IP addresses of the cluster node instances

  16. SSH to the first (as it was specified on the cluster configuration page) cluster node as cloud-admin@

    Note: If you selected to create a new VPC and connecting through a public IP address then need to edit security group attached to the database nodes. In the rule for SSH allow access from your client system IP.

  17. The welcome message will show the current initialization status of the cluster: in progress, failed, or completed.
  18. If initialization is still in progress, then wait for it to complete (this includes Oracle software installation and configuration). You will receive a broadcast message when initialization completes or fails. Cluster initialization takes 1 to 2 hours depending on configuration.

Note: for deploying FlashGrid Cluster with Terraform please refer to the following Knowledge Base article: Deploying with Terraform on AWS

Note: for deploying FlashGrid Cluster on AWS Outposts please refer to the following Knowledge Base article: Deploying Oracle RAC on AWS Outposts

Note: for deploying FlashGrid Cluster on dedicated hosts please refer to the following Knowledge Base article: Deploying on EC2 Dedicated Hosts

Note: for deploying FlashGrid Cluster with SELinux please refer to the following Knowledge Base article: How to configure SELinux mode during system deployment

4 After Deploying a Cluster

4.1 Verifying Cluster Status

On any of the cluster nodes run flashgrid-cluster command to verify that the cluster status is Good and all checks are passing.

[fg@rac1 ~]$ flashgrid-cluster
FlashGrid 18.07.10.46032 #95f2b5603f206af26482ac82386b1268b283fc3c
License: via Marketplace Subscription
Support plan: 24x7
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
FlashGrid running: OK
Clocks check: OK
Configuration check: OK
Network check: OK

Querying nodes: quorum, rac1, rac2 ...

Cluster Name: myrac
Cluster status: Good
--------------------------------------------------------------
Node    Status  ASM_Node  Storage_Node  Quorum_Node  Failgroup
--------------------------------------------------------------
rac1    Good    Yes       Yes           No           RAC1
rac2    Good    Yes       Yes           No           RAC2
racq    Good    No        No            Yes          QUORUM
--------------------------------------------------------------
--------------------------------------------------------------------------------------------------------
GroupName  Status  Mounted   Type    TotalMiB  FreeMiB  OfflineDisks  LostDisks  Resync  ReadLocal  Vote
--------------------------------------------------------------------------------------------------------
GRID       Good    AllNodes  NORMAL  12588     3376     0             0          No      Enabled    3/3
DATA       Good    AllNodes  NORMAL  2048000   2048000  0             0          No      Enabled    None
FRA        Good    AllNodes  NORMAL  1024000   1024000  0             0          No      Enabled    None
--------------------------------------------------------------------------------------------------------

4.2 OS User Accounts

During cluster initialization the following OS user accounts are created:

  • cloud-admin - the user account used to SSH to the EC2 instances with the SSH key that was selected when creating the cluster configuration. The user has sudo rights.
  • fg – can be used for running FlashGrid Storage Fabric or FlashGrid Cloud Area Network utilities. The user fg has sudo rights. The user has key-based SSH configured between all nodes of the cluster.
  • grid - Grid Infrastructure (GI) owner. GI environment variables are preconfigured.
  • oracle - Database home owner. Database environment variables, except ORACLE_SID and ORACLE_UNQNAME, are preconfigured. After creating a database, you can configure ORACLE_SID and ORACLE_UNQNAME by editing /home/oracle/.bashrc file on each database node.

Note that no passwords are configured for any users. Also, password based SSH authentication is disabled in /etc/ssh/sshd_config. Key-based authentication is recommended for better security. Creating passwords for any user is not recommended.

Users cloud-admin and fg have sudo rights and allows switching to any other user without requiring a password (which is not configured by default). Example:

sudo su - grid

Users fg, grid, and oracle have key-based SSH access configured between the nodes of the cluster. The corresponding key pairs are generated automatically during cluster initialization. For example, if you are logged in to node1 as user fg then you can SSH into node2 by simply running 'ssh node2' without entering a password or providing a key.

4.3 Finalizing Cluster Configuration

See Knowledge Base articles for performing the following steps:

  1. Creating a Database
  2. Connecting Database Clients

Note: ACFS support on RHEL may require an additional Oracle Clusterware patch. Please refer to Oracle KB129209 for ACFS patch information.

4.4 Enabling Termination Protection

If termination protection was not enabled when creating the cluster and if the cluster is for production use, then it is strongly recommended to enable termination protection:

  • Enable instance termination protection for each cluster node instance.
  • Enable termination protection for the CloudFormation stack.

4.5 Installing an Additional Database Home

In most cases manual installation of database software is not required. However, if you need to install an additional database home, then follow Oracle Database documentation for installing the database software.

4.6 Use of Anti-virus and Other Third-party software

If anti-virus software must be used, then it is recommended to configure it in a way that avoids putting any files in quarantine. Automatic quarantine of files creates risk of the cluster downtime in case of a false positive detection on a critical system file on multiple nodes of the cluster.

Any proprietary kernel modules installed by third-party software create risks to reliable operation of the system. Such proprietary kernel modules are not tested or supported by FlashGrid, Red Hat, or Oracle Linux. Proprietary kernel modules may consume kernel resources and may create instability, especially under high load. Symptoms may include kernel crashes, network disruptions, storage i/o disruptions, node evictions, and cluster brown-out. If such reliability issue is encountered and no other root cause can be readily identified, FlashGrid support reserves the right to request removal of all proprietary kernel modules before continuing investigation.

4.7 Use of Automatic Configuration Tools

Automatic configuration tools (e.g. Ansible, Salt, etc.) must be used with extra care. Incorrect modification of a critical system file (e.g. /etc/resolv.conf) on multiple cluster nodes may cause cluster downtime. Note that many critical system configuration files are protected with immutable attribute and have warnings in them. Do not remove the immutable attribute or allow automatic modification of such files unless absolutely necessary.

4.8 Security Hardening

For applying security hardening to the OS using CIS aligned security profiles, see the Knowledge Base article Security hardening of the OS during deployment.

For applying a different hardening profile, the following steps are recommended:

  1. Request FlashGrid support to review the list of required changes
  2. Back up all cluster nodes
  3. Implement the required changes on all nodes
  4. Restart the entire cluster
  5. Verify health of the cluster: sudo flashgrid-health-check
  6. In case of errors, roll back the changes or restore the nodes from backup

4.9 Convert RHEL PAYG to BYOS License

If RHEL is used, cluster instances are provisioned with a RHEL PAYG license attached to each EC2 instance. To adopt BYOS, the RHEL license must be converted as per the procedure documented by AWS, following cluster deployment.

5 Monitoring Cluster Health

The following methods of monitoring cluster health are available:

  • flashgrid-cluster utility displays status of the storage subsystem (FlashGrid Storage Fabric and ASM) and its main components. The utility can be used in monitoring scripts. It returns a non-zero value if status of the cluster is Warning or Critical.
  • flashgrid-health-check utility checks multiple items including database configuration, storage, OS kernel, config file modifications, errors in the logs, and other items that may affect health of the cluster or could help with troubleshooting. It is recommended for manual checks only.
  • FlashGrid Node Monitor service is part of the flashgrid-diags package. It provides monitoring of various system health indicators, including CPU utilization, available memory, and clocks.
  • Alerts about failures are recorded in system log and can be analyzed by 3rd-party tools.
  • Email alerts can be configured in FlashGrid Launcher and sent to one or several email addresses. See the Knowledge Base article Configuring email for FlashGrid notifications.
  • FlashGrid Node Monitor can send alerts and diagnostic uploads via HTTP/HTTPS to a remote endpoint, with support for routing through an HTTP proxy. Node Monitor detects a condition, then sends either an email alert and/or an HTTP/REST request to a monitoring/alerting service.
  • ASM disk group monitoring and alerting via Oracle Enterprise Manager.

6 Before Going Live

Before switching the cluster to live use (run commands as user fg):

  1. Confirm that only minimally required access to the cluster node instances is allowed in network security settings, and remove any unnecessary access
  2. Verify health of the cluster: flashgrid-health-check
  3. Confirm that email alerts are configured and delivered: flashgrid-node test-alerts
  4. Upload diags to FlashGrid support: flashgrid-diags upload-all
  5. Stop the cluster and back up all cluster nodes
  6. Start the cluster and do final check of the cluster health: flashgrid-health-check

7 Deleting a Cluster

To delete a cluster

  1. Disable instance termination protection for each cluster node if it was enabled.
  2. Open AWS CloudFormation Manager console.
  3. Disable termination protection for the corresponding CloudFormation stack if it was enabled.
  4. Delete the stack corresponding to the cluster.
  5. If any EBS volumes were added after deploying the cluster, those volumes must be deleted separately.
  6. If any AMI images or volume snapshots were created after deploying the cluster, those AMIs and snapshots must be deleted separately.

8 Additional Documentation

FlashGrid Cluster on AWS Knowledge Base

FlashGrid Cluster on AWS Backup Best Practices

FlashGrid Storage Fabric CLI Reference Guide

FlashGrid Cloud Area Network CLI Reference Guide

9 Technical Support

For technical help with FlashGrid Cluster please open a support request.

To expedite troubleshooting please also collect and upload diagnostic data to the secure storage used by FlashGrid support by running the following command as user fg:

flashgrid-diags upload-all

For reporting emergency type of issues that require immediate attention please also use the 24/7 telephone hotline: +1-650-641-2421 ext 7. Please note that use of the 24/7 hotline is reserved for emergency situations only.

Support Tiers and SLA details document: FlashGrid Technical Support Services.