Node panics when eth4 and eth6 network interfaces are disconnected

When the network interfaces corresponding to eth4 and eth6 go offline or manually made offline using commands such as, ifconfig ethx down, the node panics, and restarts. This is because when the private network links used for LLT heartbeat messaging are disconnected, the node gets isolated from the other nodes in the cluster and to avoid network split brain, the vxfencing module performs node membership arbitration and deliberately panics the node to avoid data corruption.

Network interfaces corresponding to eth4 and eth6 should never be disconnected as they are used as private heartbeat links among cluster nodes. (IA-26984)

The following are sample messages in the crash dump of the node that panics:

[19737.900357] LLT INFO V-14-1-10032 link 0 (eth4) node 2 inactive 
15 sec (16250505)
[19737.950354] LLT INFO V-14-1-10509 link 0 (eth4) node 2 expired
[19738.050361] LLT INFO V-14-1-10032 link 0 (eth4) node 3 inactive 
15 sec (16250505)
[19738.100361] LLT INFO V-14-1-10509 link 0 (eth4) node 3 expired
[19742.720979] VXFEN INFO V-11-1-80 RACER Node is: 0
[19742.720998] VXFEN INFO V-11-1-87 Initiating VxFen Race
[19742.720999] VXFEN INFO V-11-1-111 VxFen Pre-Race Delay: 0
[19742.721012] VXFEN INFO V-11-1-119 LEADER Node : 0 is in current 
sub-cluster
[19742.721018] VXFEN CRITICAL V-11-1-89 RACER Node lost the VxFen race
[19742.721019] VXFEN INFO V-11-1-112 VxFen Post-Race Delay: 0
[19742.721023] VXFEN NOTICE V-11-1-92 Sending LOST_RACE
[19742.721075] Kernel panic - not syncing: VXFEN CRITICAL V-11-1-20 
Local cluster node ejected from cluster to prevent potential data 
corruption.
[19742.722157] CPU: 0 PID: 8953 Comm: vxfen Kdump: loaded Tainted: P OE
 ------------ T 3.10.0-1062.9.1.el7.x86_64 #1
[19742.722486] Hardware name: Veritas NetBackup Archive 3420/X11DPU, 
BIOS 3.0c 03/27/2019
[19742.722808] Call Trace:
[19742.722965] [<ffffffffa757ac23>] dump_stack+0x19/0x1b
[19742.723129] [<ffffffffa7574967>] panic+0xe8/0x21f
[19742.723300] [<ffffffffc10668f2>] vxfen_plat_panic+0xc2/0xd0 [vxfen]
[19742.723467] [<ffffffffc1054d61>] vxfen_process_client_msg+0x6d1/0xb30
[vxfen]
[19742.723779] [<ffffffffc1055d23>] vxfen_vrfsm_cback+0x323/0x1750 [vxfen]
[19742.723947] [<ffffffffc1055a00>] ? vxfen_reconfig_msg+0x840/0x840 [vxfen]
[19742.724117] [<ffffffffc1073be8>] vrfsm_step+0x1c8/0x3a0 [vxfen]
[19742.724280] [<ffffffffc1055a00>] ? vxfen_reconfig_msg+0x840/0x840 [vxfen]
[19742.724448] [<ffffffffc1075521>] vrfsm_recv_thread+0x401/0x9b0 [vxfen]
[19742.724613] [<ffffffffc1075120>] ? vrfsm_defer_message+0x140/0x140 [vxfen]
[19742.724782] [<ffffffffc10761ee>] vxplat_lx_thread_base+0x9e/0xf0 [vxfen]
[19742.724947] [<ffffffffc1076150>] ? vxplat_assert+0x20/0x20 [vxfen]
[19742.725123] [<ffffffffa6ec61f1>] kthread+0xd1/0xe0
[19742.725282] [<ffffffffa6ec6120>] ? insert_kthread_work+0x40/0x40
[19742.725449] [<ffffffffa758dd1d>] ret_from_fork_nospec_begin+0x7/0x21
[19742.725611] [<ffffffffa6ec6120>] ? insert_kthread_work+0x40/0x40

Workaround:

Bring eth4 and eth6 online to allow the node to join the cluster properly.