Skip to main content
FlashGrid.io

How to move disk groups between clusters (Azure)

  • Updated
  • Products: FlashGrid Cluster on Azure
 

Preamble

In certain cases it is desirable to move data between two clusters by moving their data disks between the clusters. This article describes the procedure that must be followed to have consistent configuration after the disks are moved.

Further in this article, the cluster from which disk groups are moved is called the source cluster, and the cluster to which disk groups are moved is called the target cluster.

Note that some steps are different depending on whether the source and the target clusters have the same hostnames or different hostnames.

Deploy a target cluster

Deploy a target cluster without any disk groups. The GRID disk groups will be automatically created. After the deployment, the target cluster will have ASM instances, and listeners running.

Save database resource configuration

The following Clusterware resources have to be manually recreated in the target cluster: databases, instances, and services. Gather their configuration in the source cluster. Here is the list of commands that can be used for that:

  • Get a list of databases: srvctl config database
  • Get configuration for a database: srvctl config database -db <db_unique_name>
  • Get instances for a database: srvctl status database -db <db_unique-name>
  • Get services for a database: srvctl status service -db <db_unique_name>
  • Get configuration for a service: srvctl config service -db <db_unique_name> -service <service_name>
  • Get audit_file_dest directories:show spparameter audit_file_dest

Apart from database related resources, the cluster can have other resources registered with Clusterware, e.g. ACFS filesystems. Steps to recreate these resources are not considered in this article but they are essentially the same as for the database ones.

Shutdown the source cluster

Note: Make sure that the cluster status is Good before shutting it down.

Follow the maintenance article: Shutting down entire cluster

Move disks between clusters

Detach the disks from the source cluster using Azure Portal or the Azure CLI. Only the disks for the disk groups being moved should be detached. Each cluster has its own GRID disk group, so that its disks should not be moved.

Attach the disks to the target cluster. Try to attach each disk using the same LUN number that was previously used if possible.

Run: flashgrid-node reload-config on all DB and quorum nodes of the target cluster.

Mounting ASM disk groups

Mount the disk groups on each DB node of the target cluster as the grid user:

asmcmd mount <dg_name>

Check status of the disk group resources on the target cluster:

crsctl stat res -t -w 'TYPE = ora.diskgroup.type'
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.DATA.dg(ora.asmgroup)
      1        ONLINE  ONLINE       newrac1                  STABLE
      2        ONLINE  ONLINE       newrac2                  STABLE
ora.FRA.dg(ora.asmgroup)
      1        ONLINE  ONLINE       newrac1                  STABLE
      2        ONLINE  ONLINE       newrac2                  STABLE
ora.GRID.dg(ora.asmgroup)
      1        ONLINE  ONLINE       newrac1                  STABLE
      2        ONLINE  ONLINE       newrac2                  STABLE
--------------------------------------------------------------------------------

Make sure that the disk groups are online on all DB nodes of the target cluster before proceeding further.

Rename disks (same hostnames only)

Note: apply this step only if the source and target clusters use the same hostnames (domain names can be different).

Run flashgrid-cluster fix-disk-names as the grid user to generate SQL code to change disk names.

There are two possible scenarios:

1. The disks are attached to the target cluster with the same LUN numbers that they are in the source cluster.

flashgrid-cluster fix-disk-names shows that no disks should be fixed:

flashgrid-cluster fix-disk-names
FlashGrid 21.6.36.54852 #cb710355e48fe1493f2140cdfa64e5ac09940d2c
License: Active, Expires 2021-10-16
Licensee: Company_test
Support plan: Demo
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
No local disks need to be fixed

Proceed to Adding database resources.

2. All or some disks use different LUN numbers.

flashgrid-cluster fix-disk-names shows that some disks need to be renamed:

flashgrid-cluster fix-disk-names
FlashGrid 21.6.36.54852 #cb710355e48fe1493f2140cdfa64e5ac09940d2c
License: Active, Expires 2021-10-16
Licensee: Company_test
Support plan: Demo
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Disk /dev/flashgrid/racq.lun4 has invalid ASM name RACQ$LUN3 != RACQ$LUN4 and needs to be fixed
Disk /dev/flashgrid/rac2.lun4 has invalid ASM name RAC2$LUN3 != RAC2$LUN4 and needs to be fixed
Disk /dev/flashgrid/rac1.lun4 has invalid ASM name RAC1$LUN3 != RAC1$LUN4 and needs to be fixed
--------------------------------------------------------------------------------
You should mount following ASM disk group(s) in RESTRICTED mode: FRA, DATA
After that execute:
  ALTER DISKGROUP DATA RENAME DISK 'RACQ$LUN3' TO 'RACQ$LUN4';
  ALTER DISKGROUP DATA RENAME DISK 'RAC2$LUN3' TO 'RAC2$LUN4';
  ALTER DISKGROUP DATA RENAME DISK 'RAC1$LUN3' TO 'RAC1$LUN4';
================================================================================

Unmount the disk groups on each DB node of the target cluster as the grid user:

asmcmd umount <dg_name>

Mount the disk groups in restricted mode on the first DB node:

asmcmd mount --restrict <dg_name>

Connect sqlplus / as sysasm and run the SQL commands from the flashgrid-cluster fix-disk-names output, e.g.:

SQL> ALTER DISKGROUP DATA RENAME DISK 'RACQ$LUN3' TO 'RACQ$LUN4';

Diskgroup altered.
SQL> ALTER DISKGROUP DATA RENAME DISK 'RAC2$LUN3' TO 'RAC2$LUN4';

Diskgroup altered.
SQL> ALTER DISKGROUP DATA RENAME DISK 'RAC1$LUN3' TO 'RAC1$LUN4';

Diskgroup altered.

Unmount the disk group from the first DB node:

asmcmd umount <dg_name>

Mount the disk groups in normal mode on all DB nodes as grid user:

asmcmd mount <dg_name>

Replace disks (different hostnames only)

Note: apply this step only if the source and target clusters use different hostnames.

All disks of the source cluster have old failure group names which are the same as the hostnames of the source cluster. In the example below, the source hostnames are RAC1, RAC2, and RACQ whereas the target hostnames are NEWRAC1, NEWRAC2, and NEWRACQ:

[fg@newrac1 ~]$ flashgrid-dg show -G fra
FlashGrid 21.6.36.54852 #cb710355e48fe1493f2140cdfa64e5ac09940d2c
License: Active, Expires 2021-10-16
Licensee: Company_test
Support plan: Demo
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Querying nodes: newrac1, newrac2, newracq ...
----------------------------------
Disk Group:             FRA
Status:                 Good
Redundancy:             NORMAL
Total:                  32768
Free:                   24104
ReadLocal:              Enabled
Resync:                 No
Offline disks:          0
Lost disks:             0
Disk repair time:       24000h
Failgroup repair time:  24000h
Compatible ASM:         19.0.0.0.0
Compatible RDBMS:       19.0.0.0.0
Compatible ADVM:        19.0.0.0.0
----------------------------------
----------------
Node     Mounted
----------------
newrac1  Yes
newrac2  Yes
----------------
---------------------------------------------------------------------------------
FailGroup  ASM_Disk_Name  Drive                         Phys_GiB  ASM_GiB  Status
---------------------------------------------------------------------------------
RAC1       RAC1$LUN3      /dev/flashgrid/newrac1.lun3   16        16       ONLINE
RAC2       RAC2$LUN3      /dev/flashgrid/newrac2.lun3   16        16       ONLINE
RACQ       RACQ$LUN3      /dev/flashgrid/newracq.lun3   1         1        ONLINE
---------------------------------------------------------------------------------

Oracle does not support changing failure group names, so that the disks should be dropped and added back instead.

Create and attach new disks to DB and quorum nodes that are of the same size as the current disks in an ASM disk group.

Run the flashgrid-dg replace-all-disks command specifying the disks attached to the DB nodes:

flashgrid-dg replace-all-disks -G <dg_name> -d <disk_name_1> .. <disk_name_N>

For example:

[fg@newrac1 ~]$ flashgrid-dg replace-all-disks -G fra -d /dev/flashgrid/newrac[12].lun4
FlashGrid 21.6.36.54852 #cb710355e48fe1493f2140cdfa64e5ac09940d2c
License: Active, Expires 2021-10-16
Licensee: Company_test
Support plan: Demo
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Querying nodes: newrac1, newrac2, newracq ...
The following disks will replace _ALL_ disks in FRA:
    /dev/flashgrid/newrac1.lun4
   /dev/flashgrid/newrac2.lun4
Do you want to continue? (y/n):
y
Processing FRA ...
Completing this operation may take long time. Please wait...
ALTER DISKGROUP "FRA"
ADD  REGULAR FAILGROUP "NEWRAC1" DISK '/dev/flashgrid/newrac1.lun4' NAME "NEWRAC1$LUN4" SIZE 16384M
     REGULAR FAILGROUP "NEWRAC2" DISK '/dev/flashgrid/newrac2.lun4' NAME "NEWRAC2$LUN4" SIZE 16384M
DROP REGULAR DISK "RAC2$LUN3"
     REGULAR DISK "RAC1$LUN3" REBALANCE POWER 11 WAIT
Success

Wait till the Resync=No for the disk group in the flashgrid-cluster output. It may take a while to resync the disk group depending on the amount of data, disk performance characteristics, and instance parameters.

Note: Starting with Storage Fabric version 23.04, the replace-all-disks operation executes data rebalance in background. While rebalance is running in background, the flashgrid-cluster and flashgrid-dg commands may show old disks reported as "LostDisks". This will be cleared after the background operation completes.

Replace the quorum disks:

flashgrid-dg replace-disk -G <dg_name> -q <old_quorum_disk> -r <new_quorum_disk>

For example:

[fg@newrac1 ~]$ flashgrid-dg replace-disk -G FRA -q /dev/flashgrid/newracq.lun3 -r /dev/flashgrid/newracq.lun4
FlashGrid 21.6.36.54852 #cb710355e48fe1493f2140cdfa64e5ac09940d2c
License: Active, Expires 2021-10-16
Licensee: Company_test
Support plan: Demo
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Querying nodes: newrac1, newrac2, newracq ...
Replacing disk...
Completing this operation may take long time. Please wait...
ALTER DISKGROUP "FRA"
ADD  QUORUM FAILGROUP "NEWRACQ" DISK '/dev/flashgrid/newracq.lun4' NAME "NEWRACQ$LUN4"
DROP QUORUM DISK "RACQ$LUN3"
Success

At the end, the disk group will have failure groups with the hostnames of the target cluster:

[fg@newrac1 ~]$ flashgrid-dg show -G fra
FlashGrid 21.6.36.54852 #cb710355e48fe1493f2140cdfa64e5ac09940d2c
License: Active, Expires 2021-10-16
Licensee: Company_test
Support plan: Demo
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Querying nodes: newrac1, newrac2, newracq ...
----------------------------------
Disk Group:             FRA
Status:                 Good
Redundancy:             NORMAL
Total:                  32768
Free:                   24096
ReadLocal:              Enabled
Resync:                 No
Offline disks:          0
Lost disks:             0
Disk repair time:       24000h
Failgroup repair time:  24000h
Compatible ASM:         19.0.0.0.0
Compatible RDBMS:       19.0.0.0.0
Compatible ADVM:        19.0.0.0.0
----------------------------------
----------------
Node     Mounted
----------------
newrac1  Yes
newrac2  Yes
----------------
---------------------------------------------------------------------------------
FailGroup  ASM_Disk_Name  Drive                         Phys_GiB  ASM_GiB  Status
---------------------------------------------------------------------------------
NEWRAC1    NEWRAC1$LUN4   /dev/flashgrid/newrac1.lun4   16        16       ONLINE
NEWRAC2    NEWRAC2$LUN4   /dev/flashgrid/newrac2.lun4   16        16       ONLINE
NEWRACQ    NEWRACQ$LUN4   /dev/flashgrid/newracq.lun4   1         1        ONLINE
---------------------------------------------------------------------------------

Repeat steps above for all other disk groups that have been moved to the target cluster.

After that run flashgrid-cluster to confirm that there are no warnings and the cluster status is Good.

The disks from the source cluster can be removed:

Removing Disks

Adding database resources

Add CRS resources that you saved at the Save database resource configuration step, for example:

  • Adding a database: 
    srvctl add database \
  -db racdb \
  -oraclehome /u01/app/oracle/product/19.3.0/dbhome_1 \
  -spfile +DATA/RACDB/PARAMETERFILE/spfile.272.1077989921 \
  -pwfile +DATA/RACDB/PASSWORD/pwdracdb.256.1077987839 \
  -diskgroup DATA,FRA
  • Adding instances: 
    srvctl add instance \
  -db racdb \
  -instance racdb1 \
  -node newrac1 

srvctl add instance \
  -db racdb \
  -instance racdb2 \
  -node newrac2
  • Adding services: 
    srvctl add service \
  -db racdb \
  -service test_svc \
  -preferred racdb1,racdb2 \
  -pdb pdb1

srvctl add service \
  -db racdb \
  -service test2_svc \
  -preferred racdb1,racdb2 \
  -pdb pdb1
  • Create audit directories (the audit_file_dest parameter) for a database on all DB nodes: 
    mkdir -p /u01/app/oracle/admin/racdb/adump

This way you let CRS know about your databases.

Finally, start your databases:

srvctl start database -db <db_name>