Node panics when eth4 and eth6 network interfaces are disconnected
When the network interfaces corresponding to eth4 and eth6 go offline or manually made offline using commands such as, ifconfig ethx down, ip link set down ethx, or ifdown ethx, the node panics, and restarts. This is because when the private network links used for LLT heartbeat messaging are disconnected, the node gets isolated from the other nodes in the cluster and to avoid network split brain, the vxfencing module performs node membership arbitration and deliberately panics the node to avoid data corruption.
Network interfaces corresponding to eth4 and eth6 should never be disconnected as they are used as private heartbeat links among cluster nodes. (IA-26984)
The following are sample messages in the crash dump of the node that panics:
[19737.900357] LLT INFO V-14-1-10032 link 0 (eth4) node 2 inactive 15 sec (16250505) [19737.950354] LLT INFO V-14-1-10509 link 0 (eth4) node 2 expired [19738.050361] LLT INFO V-14-1-10032 link 0 (eth4) node 3 inactive 15 sec (16250505) [19738.100361] LLT INFO V-14-1-10509 link 0 (eth4) node 3 expired [19742.720979] VXFEN INFO V-11-1-80 RACER Node is: 0 [19742.720998] VXFEN INFO V-11-1-87 Initiating VxFen Race [19742.720999] VXFEN INFO V-11-1-111 VxFen Pre-Race Delay: 0 [19742.721012] VXFEN INFO V-11-1-119 LEADER Node : 0 is in current sub-cluster [19742.721018] VXFEN CRITICAL V-11-1-89 RACER Node lost the VxFen race [19742.721019] VXFEN INFO V-11-1-112 VxFen Post-Race Delay: 0 [19742.721023] VXFEN NOTICE V-11-1-92 Sending LOST_RACE [19742.721075] Kernel panic - not syncing: VXFEN CRITICAL V-11-1-20 Local cluster node ejected from cluster to prevent potential data corruption. [19742.722157] CPU: 0 PID: 8953 Comm: vxfen Kdump: loaded Tainted: P OE ------------ T 3.10.0-1062.9.1.el7.x86_64 #1 [19742.722486] Hardware name: Veritas NetBackup Archive 3420/X11DPU, BIOS 3.0c 03/27/2019 [19742.722808] Call Trace: [19742.722965] [<ffffffffa757ac23>] dump_stack+0x19/0x1b [19742.723129] [<ffffffffa7574967>] panic+0xe8/0x21f [19742.723300] [<ffffffffc10668f2>] vxfen_plat_panic+0xc2/0xd0 [vxfen] [19742.723467] [<ffffffffc1054d61>] vxfen_process_client_msg+0x6d1/0xb30 [vxfen] [19742.723779] [<ffffffffc1055d23>] vxfen_vrfsm_cback+0x323/0x1750 [vxfen] [19742.723947] [<ffffffffc1055a00>] ? vxfen_reconfig_msg+0x840/0x840 [vxfen] [19742.724117] [<ffffffffc1073be8>] vrfsm_step+0x1c8/0x3a0 [vxfen] [19742.724280] [<ffffffffc1055a00>] ? vxfen_reconfig_msg+0x840/0x840 [vxfen] [19742.724448] [<ffffffffc1075521>] vrfsm_recv_thread+0x401/0x9b0 [vxfen] [19742.724613] [<ffffffffc1075120>] ? vrfsm_defer_message+0x140/0x140 [vxfen] [19742.724782] [<ffffffffc10761ee>] vxplat_lx_thread_base+0x9e/0xf0 [vxfen] [19742.724947] [<ffffffffc1076150>] ? vxplat_assert+0x20/0x20 [vxfen] [19742.725123] [<ffffffffa6ec61f1>] kthread+0xd1/0xe0 [19742.725282] [<ffffffffa6ec6120>] ? insert_kthread_work+0x40/0x40 [19742.725449] [<ffffffffa758dd1d>] ret_from_fork_nospec_begin+0x7/0x21 [19742.725611] [<ffffffffa6ec6120>] ? insert_kthread_work+0x40/0x40
Workaround:
Bring eth4 and eth6 online to allow the node to join the cluster properly.
You can also use the ifup ethx command if any of the above mentioned commands are used to bring down the interface. If the physical cable is disconnected, then reconnect it.