FlashGrid Cluster on Azure. Deployment Guide – FlashGrid Help Center

rev. 26.05

1 Overview
    1.1 Key Components
    1.2 Infrastructure-as-Code Deployment
2 Prerequisites
3 Deploying a Cluster
4 After Deploying a Cluster
    4.1 Verifying Cluster Status
    4.2 OS User Accounts
    4.3 Finalizing Cluster Configuration
    4.4 Adding a Protection Lock for the Cluster
    4.5 Installing an Additional Database Home
    4.6 Use of Anti-virus and Other Third-party Software
    4.7 Use of Automatic Configuration Tools
    4.8 Security Hardening
    4.9 Convert RHEL PAYG to BYOS License
5 Monitoring Cluster Health
6 Before Going Live
7 Deleting a Cluster
8 Additional Documentation
9 Technical Support

1 Overview

FlashGrid Cluster is a virtual clustered appliance that enables deployment of Oracle databases with Oracle RAC or Failover HA on native Azure VMs for mission-critical workloads. Oracle RAC nodes can be distributed across Azure Availability Zones to support database uptime of 99.99% or higher, depending on the deployment architecture and availability requirements. Deployment is fully automated using Infrastructure-as-Code and typically completes in 30 to 90 minutes. FlashGrid provides 24/7 technical support covering the complete infrastructure stack used by the cluster. First introduced on Azure in 2017, FlashGrid Cluster is available in all Azure regions, including Azure Government and Azure China.

Detailed information about FlashGrid Cluster architecture for Oracle databases on Azure is available on the following pages:

1.1 Key Components

Key components of FlashGrid Cluster on Azure:

FlashGrid Storage Fabric software
FlashGrid Cloud Area Network software
Oracle Database: 26ai, 19c, 12.2.0.1, 12.1.0.2, or 11.2.0.4
Oracle Grid Infrastructure: 26ai or 19c
Operating System:
- Oracle Linux: 8 (UEKR7) or 9 (UEKR7)
- Red Hat Enterprise Linux (RHEL): 8 or 9
Azure VMs:
- General purpose: Dasv7, Dsv6, Dasv6, Dsv5
- Compute optimized: FXmsv2
- Memory optimized: Easv7, Esv6, Easv6, Ebsv5, Esv5, Mbsv3, Msv3
Disks: Premium SSD or Premium SSD v2

1.2 Infrastructure-as-Code Deployment

FlashGrid Cluster is delivered as an Azure Resource Manager template that automates configuration of multiple components required for a database cluster. FlashGrid Launcher is an online tool that simplifies the deployment process by guiding through the cluster configuration parameters and generating Azure Resource Manager templates.

2 Prerequisites

The following prerequisites are required for automated deployment of a FlashGrid Cluster:

Azure Storage Blob Container with Oracle installation files that will be downloaded to the cluster nodes during cluster initialization. FlashGrid Launcher will show you the list of files that must be placed in the Storage Container. The files in the Storage Container must be accessible from the VMs in the target VNet. For configuring access permission options, see the Knowledge Base article Uploading Oracle Installation Files to Azure Storage Container.
Microsoft.Storage service endpoint configured for the VNet. Having the storage service endpoint allows access to the storage container from the VMs. If Microsoft.Storage service endpoint is not added, and public IPs not assigned then cluster initialization will fail because downloading Oracle files from the VMs will not be possible.
Azure subscription with sufficient quotas for creating the required number and type of VMs and sufficient number and size of Premium Managed Disks.
SSH key pair that will be used for accessing the VMs. Use of passwords instead of the key pair is not supported. To create a new key pair use ssh-keygen in Linux or puttygen in Windows. In the FlashGrid Launcher tool you will need to provide the public key that will be placed on the VMs. Example of a valid public key pair format:
```
ssh-rsa <PublicKeyBody>
```
Keep blank if using Azure managed SSH keys for VMs access.
Properly configured Network Security Group (NSG) when deploying in an existing VNet. You have a choice of attaching an NSG to the VMs or using the NSG attached to the subnet. The following ports should be open:
- Inbound and Outbound: All traffic between the cluster nodes.
- Inbound: TCP ports 1521, 1522 for SCAN and Local Listener access to the database nodes from app servers and other database clients. These are default port numbers that can be changed in the FlashGrid Launcher tool.
- Inbound: TCP port 22 for SSH access to the cluster nodes
- Inbound: TCP port 5901 if you choose to use VNC for creating a database using DBCA in GUI mode with direct connection (vs. SSH tunnel)
- Inbound access to the ports listed above must be allowed only from those application security groups or IP ranges that require such access. Do not configure Any or 0.0.0.0/0 as allowed sources.
FlashGrid recommends configuring the NSG rules by using an Application Security Group (ASG) for the cluster node VMs. You can configure one ASG shared by clusters or a separate ASG for each cluster.

Note: for deploying FlashGrid Cluster on dedicated hosts please refer to the Knowledge Base article Deploying on dedicated hosts (Azure).

3 Deploying a Cluster

The FlashGrid Launcher tool simplifies deployment of database clusters on Azure by automating the following tasks:

Creating cloud infrastructure: VMs, storage, and optionally network
Installing and configuring FlashGrid Cloud Area Network
Installing and configuring FlashGrid Storage Fabric
Installing, configuring, and patching Oracle Grid Infrastructure
Installing and patching Oracle Database software
Creating ASM disk groups

To create a cluster

Open FlashGrid Launcher tool with one of the standard configurations:
- Oracle RAC: https://www.flashgrid.io/products/flashgrid-for-oracle-rac-on-azure
- Oracle Single-Instance with failover HA: https://www.flashgrid.io/products/flashgrid-for-oracle-failover-ha-on-azure/
or, if you have a custom configuration file, upload it at https://latest.cloudprov.flashgrid.io/
Configure parameters of the cluster
Click Validate Configuration button
If verification passes then click Launch Cluster button, which will take you to Azure Resource Manager
Select Resource group - Create new. By having the cluster in a separate resource group, you can later delete the entire cluster by simply deleting the resource group.
Enter a name for the new resource group that will contain the cluster. A name matching the cluster name is recommended.
Select your target location (region)
If you did not provide an SSH key, select Use existing key stored in Azure in SSH public key source and specify a stored key.
Check 'I agree to the terms and conditions state above'
Click Purchase
Open list of Notifications (bell icon) and click 'Deployment in progress…'
Wait until the deployment status changes to Succeeded
If the deployment fails:
1. Check for the cause of the failure in the Operation details
2. Correct the cause of the error
3. Delete the failed resource group
4. Repeat the steps for creating a new resource group
SSH to the first (as it was specified on the cluster configuration page) cluster node VM as user az-admin@

Note: If you selected to create a new VNet and connecting through a public IP address then need to edit the network security group attached to the database nodes. In the Azure Virtual Machine settings select Networking, and for the allow-tcp22 inbound port rule set Source to your client system IP.
The welcome message will show the current initialization status of the cluster: in progress, failed, or completed.
If initialization is still in progress, then wait for it to complete (this includes Oracle software installation and configuration). You will receive a broadcast message when initialization completes or fails. Cluster initialization takes 1 to 2 hours for RAC/HA cluster or 30 to 40 minutes for a single-instance with no HA.

Note: for deploying FlashGrid Cluster with SELinux please refer to the Knowledge Base article: How to configure SELinux mode during system deployment.

4 After Deploying a Cluster

4.1 Verifying Cluster Status

On any of the cluster nodes run flashgrid-cluster command to verify that the cluster status is Good and all checks are passing.

[fg@rac1 ~]$ flashgrid-cluster
FlashGrid 18.07.15.48564 #95f2b5603f206af26482ac82386b1268b283fc3c
License: via Marketplace Subscription
Support plan: 24x7
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
FlashGrid running: OK
Clocks check: OK
Configuration check: OK
Network check: OK

Querying nodes: quorum, rac1, rac2 ...

Cluster Name: myrac
Cluster status: Good
--------------------------------------------------------------
Node    Status  ASM_Node  Storage_Node  Quorum_Node  Failgroup
--------------------------------------------------------------
rac1    Good    Yes       Yes           No           RAC1
rac2    Good    Yes       Yes           No           RAC2
racq    Good    No        No            Yes          QUORUM
--------------------------------------------------------------
--------------------------------------------------------------------------------------------------------
GroupName  Status  Mounted   Type    TotalMiB  FreeMiB  OfflineDisks  LostDisks  Resync  ReadLocal  Vote
--------------------------------------------------------------------------------------------------------
GRID       Good    AllNodes  NORMAL  12588     3376     0             0          No      Enabled    3/3
DATA       Good    AllNodes  NORMAL  2048000   2048000  0             0          No      Enabled    None
FRA        Good    AllNodes  NORMAL  1024000   1024000  0             0          No      Enabled    None
--------------------------------------------------------------------------------------------------------

4.2 OS User Accounts

During cluster initialization the following OS user accounts are created:

az-admin - the user account used to SSH to the VMs with the SSH key that was selected when creating the cluster configuration. The user has sudo rights.
fg - can be used for running FlashGrid Storage Fabric or FlashGrid Cloud Area Network utilities. The user has sudo rights. The user has key-based SSH configured between all nodes of the cluster.
grid - Grid Infrastructure (GI) owner. GI environment variables are preconfigured. The user has key-based SSH configured between all database nodes of the cluster.
oracle - Database home owner. Database environment variables, except ORACLE_SID and ORACLE_UNQNAME, are preconfigured. After creating a database, you can configure ORACLE_SID and ORACLE_UNQNAME by editing /home/oracle/.bashrc file on each database node. The user has key-based SSH configured between all database nodes of the cluster.

Note that no passwords are configured for any users. Also, password based SSH authentication is disabled in /etc/ssh/sshd_config. Key-based authentication is recommended for better security. Creating passwords for any user is not recommended.

Users az-admin and fg have sudo rights and allows switching to any other user without requiring a password (which is not configured by default). Example:

sudo su - grid

Users fg, grid, and oracle have key-based SSH access configured between the nodes of the cluster. The corresponding key pairs are generated automatically during cluster initialization. For example, if you are logged in to node1 as user fg then you can SSH into node2 by simply running 'ssh node2' without entering a password or providing a key.

4.3 Finalizing Cluster Configuration

See knowledge base articles for performing the following steps:

Note: ACFS support on RHEL may require an additional Oracle Clusterware patch. Please refer to Oracle KB129209 for ACFS patch information.

4.4 Adding a Protection Lock for the Cluster

It is strongly recommended to add a lock to the cluster resource group to protect it against accidental deletion or modification.

4.5 Installing an Additional Database Home

In most cases manual installation of database software is not required. However, if you need to install an additional database home, then follow Oracle Database documentation for installing the database software.

4.6 Use of Anti-virus and Other Third-party Software

If anti-virus software must be used, then it is recommended to configure it in a way that avoids putting any files in quarantine. Automatic quarantine of files creates risk of the cluster downtime in case of a false positive detection on a critical system file on multiple nodes of the cluster.

Any proprietary kernel modules installed by third-party software create risks to reliable operation of the system. Such proprietary kernel modules are not tested or supported by FlashGrid, Red Hat, or Oracle Linux. Proprietary kernel modules may consume kernel resources and may create instability, especially under high load. Symptoms may include kernel crashes, network disruptions, storage i/o disruptions, node evictions, and cluster brown-out. If such reliability issue is encountered and no other root cause can be readily identified, FlashGrid support reserves the right to request removal of all proprietary kernel modules before continuing investigation.

4.7 Use of Automatic Configuration Tools

Automatic configuration tools (e.g. Ansible, Salt, etc.) must be used with extra care. Incorrect modification of a critical system file (e.g. /etc/resolv.conf) on multiple cluster nodes may cause cluster downtime. Note that many critical system configuration files are protected with immutable attribute and have warnings in them. Do not remove the immutable attribute or allow automatic modification of such files unless absolutely necessary.

4.8 Security Hardening

For applying security hardening to the OS using CIS aligned security profiles, see the Knowledge Base article Security hardening of the OS during deployment.

For applying a different hardening profile, the following steps are recommended:

Request FlashGrid support to review the list of required changes.
Back up all cluster nodes.
Implement the required changes on all nodes.
Restart the entire cluster.
Verify health of the cluster as user fg:
```
flashgrid-health-check
```
In case of errors, roll back the changes or restore the nodes from backup.

4.9 Convert RHEL PAYG to BYOS License

If RHEL is used, cluster VM instances are provisioned with a RHEL PAYG license attached to each Azure VM instance. To adopt BYOS, the RHEL license must be converted as per the procedure documented by Azure, following cluster deployment.

5 Monitoring Cluster Health

The following methods of monitoring cluster health are available:

flashgrid-cluster utility displays status of the storage subsystem (FlashGrid Storage Fabric and ASM) and its main components. The utility can be used in monitoring scripts. It returns a non-zero value if status of the cluster is Warning or Critical.
flashgrid-health-check utility checks multiple items including database configuration, storage, OS kernel, config file modifications, errors in the logs, and other items that may affect health of the cluster or could help with troubleshooting. It is recommended for manual checks only.
FlashGrid Node Monitor service is part of the flashgrid-diags package. It provides monitoring of various system health indicators, including CPU utilization, available memory, and clocks.
Alerts about failures are recorded in system log and can be analyzed by 3rd-party tools.
Email alerts can be configured in FlashGrid Launcher and sent to one or several email addresses. See the Knowledge Base article Configuring email for FlashGrid notifications.
FlashGrid Node Monitor can send alerts and diagnostic uploads via HTTP/HTTPS to a remote endpoint, with support for routing through an HTTP proxy. Node Monitor detects a condition, then sends either an email alert and/or an HTTP/REST request to a monitoring/alerting service.
ASM disk group monitoring and alerting via Oracle Enterprise Manager.

6 Before Going Live

Before switching the cluster to live use (run commands as user fg):

Confirm that only minimally required access to the cluster node instances is allowed in network security settings, and remove any unnecessary access
Verify health of the cluster: flashgrid-health-check
Confirm that email alerts are configured and delivered: flashgrid-node test-alerts
Upload diags to FlashGrid support: flashgrid-diags upload-all
Stop the cluster and back up all cluster nodes.
Start the cluster and do final check of the cluster health: flashgrid-health-check

7 Deleting a Cluster

To delete a cluster

Delete any protection lock(s) for the resource group
Delete the resource group corresponding to the cluster

8 Additional Documentation

FlashGrid Cluster on Azure Knowledge Base

FlashGrid Cluster on Azure Backup Best Practices

FlashGrid Storage Fabric CLI Reference Guide

FlashGrid Cloud Area Network CLI Reference Guide

9 Technical Support

For technical help with FlashGrid Cluster please open a support request.

To expedite troubleshooting please also collect and upload diagnostic data to the secure storage used by FlashGrid support by running the following command as user fg:

flashgrid-diags upload-all

For reporting emergency type of issues that require immediate attention please also use the 24/7 telephone hotline: +1-650-641-2421 ext 7. Please note that use of the 24/7 hotline is reserved for emergency situations only.

Support Tiers and SLA details document: FlashGrid Technical Support Services.

Table of Contents