Extending CLAN connectivity across two or more clusters may be needed in certain deployment scenarios where transparent layer 2 connectivity between the clusters is required. A typical scenario is for data replication (GoldenGate, SharePlex, DBvisit) using HA VIP for the replication service failover between cluster nodes. In such scenarios, the connectivity is required only on the fg-pub (192.168.1.X) virtual subnet.
Procedure for deploying two or more clusters with CLAN connectivity between them
- Before deploying clusters that need CLAN extended between them, add and customize the parameters below in the cluster configuration files (used by FlashGrid Launcher or its REST API). The IP addresses must be non-overlapping between all clusters. Additionally, different sets of hostnames must be used in different clusters.
[clan] first_node_id = 11 # Recommended values are 11, 21, 31,… With default settings the node IPs will start from 192.168.1.1 first_vip = 111 # Recommended values are 111, 121, 131,… With default settings the VIP addresses will be 192.168.1.11, 192.168.1.12, 192.168.1.13 scans = [211, 212, 213] # Recommended values are [211, 212, 213], [221, 222, 223], [231, 232, 233] ,… With default settings the SCAN VIP addresses will be 192.168.1.21, 192.168.1.22, 192.168.1.23
-
Deploy all clusters.
-
The clusters must be on the same VPC/VNet or have a connection between the VPCs/VNets through peering or VPN.
-
The clusters should have IP communication open on UDP port 4801 for fg-pub subnet.
- Different sets of hostnames must be used in different clusters.
-
-
Create a new
hostsfile and place it as/etc/hostson each node of each cluster, including quorum nodes. The file must contain a record for each node of each cluster in the following format:<Host VPC or VNet IP> <hostname>-ext.<domainname> <hostname>-ext -
Create and deploy a new shared CLAN cfg file:
- Copy the nodes values from the /etc/flashgrid-clan.cfg file of each cluster and merge them together. The resulting nodes value must be a list of all the nodes of all clusters.
- Create a new flashgrid-clan.cfg file by taking /etc/flashgrid-clan.cfg file from any cluster and replacing the nodes value with the merged value.
- Place the new file as /etc/flashgrid-clan.cfg on each node of each cluster, including quorum nodes.
- Deploy the new configuration. On each node of each cluster run
sudo flashgrid-clan-cfg deploy-config-local; sudo systemctl restart flashgrid-clan.service - Ensure that the clusters are connected. Ping each node of every cluster using 192.168.1.X IP addresses.
Changing MTU size
Note: This functionality is available starting with CLAN 21.04
By default, CLAN virtual NICs use the maximum Maximum Transmission Unit (MTU) size supported by the NIC minus 50 bytes. However, in certain scenarios, a smaller MTU might need to be configured if the network between clusters has a smaller maximum MTU. For example, AWS Transit Gateway imposes an 8500 bytes limit for the MTU size: https://docs.aws.amazon.com/vpc/latest/tgw/what-is-transit-gateway.html. In this case, the maximum MTU for CLAN interfaces must be 8450 (= 8500 - 50). If MTU for CLAN is not reduced accordingly, some network packets between the clusters will be lost.
Below is an example of changing MTU for fg-pub virtual NIC:
1. Edit /etc/flashgrid-clan.cfg on the first node of each cluster: add a new MTU size to clan.roles for fg-pub interface:
'database': {'direct': {'max_bw': '40%', 'min_bw': '5%', 'prio': 2},
'direct_iscsi': {'max_bw': '70%', 'min_bw': '15%', 'prio': 1},
'interface': 'eth0',
'root_bw': 'auto-aws',
'txq': 'auto',
'upload_cfg': True,
'vifs': {'fg-priv': {'max_bw': '70%',
'min_bw': '15%',
'peers': ['database', 'db-only']},
'fg-pub': {'max_bw': '15%',
'min_bw': '5%',
'mtu': 8450,
'peers': ['database',
'db-only',
'storage',
'quorum',
'client']},
2. Deploy the new configuration and restart FlashGrid CLAN service:
sudo flashgrid-clan-cfg deploy-config -f sudo systemctl restart flashgrid-clan.service
3. Check that the new MTU has been set:
$ ifconfig fg-pub | grep mtu
fg-pub: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 8450