Adjustable EBS and network bandwidth

Some EC2 instance types (e.g. R8i/R8a/M8i/M8a/X8i/X8aedz) allow adjusting bandwidth allocation between EBS and network, increasing one by decreasing the other. This article provides necessary steps to configure bandwidth in FlashGrid Cluster environments.

Supported bandwidth weighting options

At the time of writing this article, the following options are supported.

• default - uses the standard bandwidth configuration for your EC2 instance
• vpc-1 - increases bandwidth for network, and decreases bandwidth for EBS
• ebs-1 - increases bandwidth for EBS, and decreases bandwidth for network

Please consult the AWS documentation for the up-to-date list of options.

Possible use cases

• FlashGrid Cluster in standard (2-node/3-node) configurations may benefit from higher EBS bandwidth on database nodes if storage throughput (MBPS) is confirmed to create a performance bottleneck.
• FlashGrid Cluster with separate storage nodes will benefit from higher network bandwidth on database nodes.

Steps to configure bandwidth for existing EC2 instances

Preparation

1. Review current bandwidth settings for your EC2 instances.

2. Determine whether EBS or networking performance needs to be changed. Consult AWS documentation to see the impact of bandwidth settings on EBS, and network. Contact FlashGrid support if any assistance is needed.

   The following example demonstrates network and EBS bandwidth of the r8i.2xlarge EC2 instance type for various bandwidth weighting configurations: default, vpc-1, ebs-1.

   EC2 instance type	Baseline network bandwidth (Gbps)			Baseline EBS bandwidth (Gbps)
   	default	vpc-1	ebs-1	default	vpc-1	ebs-1
   r8i.2xlarge	3.75	4.688	3.125	2.5	1.562	3.125

Change CLAN configuration

Perform the steps below on the first database node only:

1. Modify /etc/flashgrid-clan.cfg setting the new bandwidth for database nodes in clan.roles:

   'database': {'direct': {'max_bw': '40%', 'min_bw': '5%', 'prio': 2},
                'direct_iscsi': {'max_bw': '70%', 'min_bw': '15%', 'prio': 1},
                'interface': 'eth0',
                'root_bw': 'auto-aws',

   Change root_bw from 'auto-aws' to the new baseline network bandwidth in Mbps, and add the mbit suffix.

   For example, change the default line 'root_bw': 'auto-aws', to 'root_bw': '3125mbit', for the ebs-1 bandwidth weighting of r8i.2xlarge.

   Notes:

   • AWS documentation shows bandwidth settings in Gbps whereas CLAN uses Mbps. Multiply the AWS documentation value by 1,000 to get the corresponding CLAN value, and add the mbit suffix.
   • Some EC2 instance types have both baseline and burst bandwidth values. Always use the baseline value.
   • Only for network bandwidths from 30Gbps to 40Gbps, use value '30000mbit' in the CLAN config. Any bandwidths lower or equal to 30Gbps, or more than 40Gbps are set as-is in the CLAN config following the notes above. For example, for network bandwidths 35Gbps, 25Gbps, and 45Gbps set the CLAN config value to '30000mbit', '25000mbit', and '45000mbit' correspondingly.
   • In case of a configuration with separate storage nodes, change root_bw for the db-only role.

2. Deploy CLAN configuration:

   # flashgrid-clan-cfg deploy-config -f

    

   The output must show - "Reloading CLAN service" on all nodes and "deployment successful" on all other nodes.

Change EC2 instance configuration

Perform the following steps on each database node, one node at a time:

1. Power off the node
2. Configure bandwidth weighting using AWS Console or CLI
3. Start the EC2 Instance using AWS console.

4. Before updating the next node, wait until the node boots up, all disks are back online, and resync operations complete on all disk groups. All disk groups must have zero offline disks and Resync = No before it is safe to proceed to the next node.

   flashgrid-cluster

Known restrictions

• Need to manually update CLAN bandwidth settings after Resizing cluster nodes. Please follow Steps to configure bandwidth for existing EC2 instances, and use the network bandwidth value corresponding to the new EC2 instance type and bandwidth weighting configuration in use. In case the new EC2 instance type does not support bandwidth weighting, change 'root_bw': 'some-value', to 'root_bw': 'auto-aws',.