Virtual machines in Azure can support a larger MTU than the 1,500-byte default (see Azure document Configure Maximum Transmission Unit (MTU) for virtual machines in Azure). To minimize fragmentation overhead and improve performance, FlashGrid Cluster for Oracle RAC (Azure) and FlashGrid Server for Oracle Database (Azure) deployments are configured to use MTU=3900 on systems deployed from version 23.08 onwards.
TCP protocol uses Path MTU Discovery to determine the smallest supported MTU for each TCP connection. All Azure and third-party end-points that use TCP protocol are supposed to be compatible with the larger MTU setting on the database server side.
However, FlashGrid has seen Azure NetApp Files (ANF) service experiencing blockage for unknown reason. All FlashGrid customers who also need to use ANF are advised to implement a workaround by adding a route-based setting for the ANF endpoints with MTU=1400. (The smaller than default setting is recommended as a further precaution.)
Required information
Identify the default route on your server(s). This can be found by running ip route show default In the example below, the default route is 10.100.0.1
[az-admin@rac1 ~]$ ip route show default
default via 10.100.0.1 dev eth0 proto dhcp src 10.100.0.6 metric 100
Then identify the IP address and subnet mask of the remote system. This might be found by identifying a single IP address that has been specified in a mount command:
mount -t nfs -o rw,hard,rsize=262144,wsize=262144,vers=3,tcp 10.100.14.132:/nfsvol01 /mnt/nfsvol01
From the above, the IP of the remote system is 10.100.14.132, and as it is a single IP address the subnet mask is /32, so the information you will use here is 10.100.14.132/32
Alternatively, you may have a block of IP address that are used for multiple mounts, such as 10.100.14.129/30
Applying the change
Begin by applying the change to one server (this action can be performed with servers online and will not impact Oracle services). Replace the IP destination and default route with your own values:
ip route add DESTINATION/SUBNET_MASK via DEFAULT_ROUTE mtu lock 1400
using example details from above:
ip route add 10.100.14.132/32 via 10.100.0.1 mtu lock 1400
And then test to see if issue has been resolved.
Once that is confirmed as working, run the above command on all remaining servers in the environment.
To make change persistent (i.e. survive reboots), run the following, ensuring the quotes are included as shown. Replace the IP destination and default route with your own values:
nmcli connection modify "System eth0" +ipv4.routes "DESTINATION/SUBNET_MASK DEFAULT_ROUTE lock-mtu=true mtu=1400"
using example details from above:
nmcli connection modify "System eth0" +ipv4.routes "10.100.14.132/32 10.100.0.1 lock-mtu=true mtu=1400"
To confirm a change has been applied, run the following command
ip route show | grep lock
10.100.14.132 via 10.100.0.1 dev eth0 mtu lock 1400