Adding a database node to a FlashGrid Cluster on Azure – FlashGrid Help Center

Note: Adding a node to an existing FlashGrid Cluster is supported only in configurations with separate "database-only" and "storage-only" nodes. It is not supported with standard 2-node or 3-node clusters that have data disks attached to database nodes.

Before You Start

FlashGrid Cluster software must be on version 24.1.108 or newer
All cluster nodes must be online
Cluster status reported by flashgrid-cluster command must be Good
Select a name for the new database node. It will be used as the VM name and the host name. Further in this article it will be referred to as <new db node>.
Request a new license file for the new node from FlashGrid Support.

Steps for Adding a New DB Node

1) Create a new VM

Create a snapshot from another database node of the <existing db node>-root volume, choose "Full" snapshot type, and "Disable public and private access" in the Networking options.
Locate the snapshot and create a disk named <new db node>-root . If using availability zones then make sure to create the disk in the desired availability zone.

Create a new AN-enabled NIC with a specific name, VNet, subnet, and NSG in a chosen resource group and statically allocated IP. Accelerated-networking option must be set to True.

 az network nic create \
 --resource-group <resource group name> \
  --name <new db node>-nic1 \
  --vnet-name <VNet name>  \
  --subnet <subnet name> \
  --accelerated-networking true \
  --network-security-group <network security group> \
  --private-ip-address <private ip address of a new db node>

Create a VM from the recently created OS disk (<new db node>-root). Indicate the resource group, VM name, plan details, OS disk name, OS type, size, previously created NIC name, and either availability zone number or availability set name.
```
az vm create \
  --resource-group <resource group name>  \
  --name <new db node> \
  --plan-name <plan name> \
  --plan-product flashgrid-skycluster \
  --plan-publisher flashgrid-inc  \
  --attach-os-disk <new db node>-root  \
  --os-type Linux \
  --size <VM type, similar to others> \
  --nics <new db node>-nic1 \
 {--zone <zone id> | --availability-set <availability set name> }
```
Use the same plan name that is used for other VMs in the cluster. To retrieve plan details from an existing VM: az vm show --name <existing VM> --resource-group <resource group name> --query plan

2) Create and attach a software disk to the new VM

Create a disk for Oracle software that matches LUN 0 disk on other database nodes. For example, the size should be 102G and the performance tier should be P10.
Attach a disk as LUN 0.

3) Add the new VM to the FlashGrid Cluster

Note: The new VM will temporarily have the same hostname as the original VM from which we cloned the OS disk.

Connect to the new node as root user and run

# flashgrid-add-node-from-clone <new db node>

Wait until the command output is successful.

4) Place the license file on the new node as /etc/flashgrid-license

5) On the new node create a logical volume and file system for Oracle software homes.

Connect as root user and run

pvcreate /dev/lun0
vgcreate fgvg /dev/lun0
lvcreate -n u01 -L 100g fgvg
mkfs.xfs /dev/fgvg/u01
mount /u01

6) Reboot the new node

# reboot

7) Add the new database node to the Clusterware

Make sure to have an up-to-date backup of Oracle Cluster Registry (OCR) prior to adding a node

As root user, create necessary directories on the new node

for i in /u01/app/oraInventory /u01/app/grid/diag /u01/app/19.3.0/grid /u01
do
mkdir -p $i
chown -R grid:oinstall $i
chmod -R 775 $i
done

for i in  /u01/app/oracle/product/19.3.0/dbhome_1  /u01/app/oracle/diag /u01/app/oracle
do
mkdir -p $i
chown -R oracle:oinstall $i
chmod -R 775 $i
done

Run the command below as grid user and as oracle user on all database nodes
```
ssh-keyscan -t ecdsa <new db node's hostname> >> ~/.ssh/known_hosts
```
Note: If you omit this command, you will encounter a PRVG-2019 error message while running addnode.sh script from the corresponding user (grid or oracle)
(skip this step if the cluster was deployed with FlashGrid Launcher 24.05 or newer) Connect to the first db node using the root user credentials and execute the following commands:
```
# systemctl stop flashgrid-gridhome-hb.service;
# systemctl disable flashgrid-gridhome-hb.service
# rm -rf /u01/app/19.3.0/grid/.fg/mon_file
```

Connect to the first db node as grid user and run the addnode.sh script from the $GI_HOME/addnode directory with following parameters:

$ /u01/app/19.3.0/grid/addnode/addnode.sh -silent "CLUSTER_NEW_NODES={<new db node>}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={<new db node>-vip}" -ignoreSysPrereqs -ignorePrereq

...
As a root user, execute the following script(s):
  1. /u01/app/19.3.0/grid/root.sh

Execute /u01/app/19.3.0/grid/root.sh on the following nodes:
[<new db node>]
...

As root, run the $GI_HOME/root.sh script on the new db node and wait until it finishes. Please note that it may take some time to complete.
```
# /u01/app/19.3.0/grid/root.sh
```
Start ohasd.service on the new db node
```
# systemctl start ohasd.service
```

Connect to the first db node as oracle user and run addnode.sh script from $RDBMS_HOME/addnode with the following parameters

$ /u01/app/oracle/product/19.3.0/dbhome_1/addnode/addnode.sh -silent "CLUSTER_NEW_NODES={<new db node>}"

...
As a root user, execute the following script(s):
  1. /u01/app/oracle/product/19.3.0/dbhome_1/root.sh

Execute /u01/app/oracle/product/19.3.0/dbhome_1/root.sh on the following nodes:
[<new db node>]
...

As root, run $RDBMS_HOME/root.sh script on the new db node
```
# /u01/app/oracle/product/19.3.0/dbhome_1/root.sh
```
As oracle user on the new db node, create tnsnames.ora file under $RDBMS_HOME/network/admin and add the following entry
```
$ cat /u01/app/oracle/product/19.3.0/dbhome_1/network/admin/tnsnames.ora

DONOTDELETE,NODEFQDN =
  (ADDRESS = (PROTOCOL = TCP)(Host = <new db node's FQDN>)(Port = 1522))
    
```
You may need to replace the port number 1522 if you are using a different local listener port number.
(skip this step if the cluster was deployed with FlashGrid Launcher 24.05 or newer) Connect to the first db node as root and run the following commands:
```
# systemctl enable flashgrid-gridhome-hb.service;
# systemctl start flashgrid-gridhome-hb.service
```

8) On the client-side DNS server, add the following record(s)

<new db node's FQDN> <VNet IP of the new node>

In case of using SCAN in connection strings, in addition to the above entry, add IP address of the new db node to the SCAN address record:

<SCAN name>  <VNet IP of the new node>

9) Update the client's connection strings

In case the client's connection string is in a long format that includes all database hostnames, it is necessary to modify the connection string and include the new node's hostname. Please see the sample configuration here.

10) Add new database instance(s) and update the corresponding database services

Use DBCA as oracle user on the first database node to add new database instance on the new node (repeat this step for each database):

dbca -silent -addInstance -nodeName <new db node> -gdbName <gdb name> [-instanceName <instance name> -sysDBAUserName sysdba -sysDBAPassword <password>]

If you have database services, please make sure that you update them accordingly

11) Confirm that the node was added successfully to the cluster

Verify the new node's CRS membership using the crsctl command
```
# crsctl status res -t
```
Check status of the new node and wait until resyncing operations are complete on all disk groups. All disk groups must have zero offline disks and Resync = No
```
$ flashgrid-cluster
```

12) Upload diags to FlashGrid support

Note: After applying any patch to GI or Database homes, you need to set group write permission for the following file on the newly added database node. Otherwise, subsequent patching may show a "permission denied" error. The file is created on the new node after the first patching.

chmod g+w /u01/app/oraInventory/ContentsXML/oui-patch.xml